Nov 01 08:31:25.909823 ip-10-0-134-93 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 08:31:26.315377 ip-10-0-134-93 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:31:26.315377 ip-10-0-134-93 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 08:31:26.315377 ip-10-0-134-93 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:31:26.315377 ip-10-0-134-93 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:31:26.322889 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322807    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.322889 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322886    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.322889 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322891    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322894    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322905    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322909    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322912    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322914    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322917    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322920    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322923    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322925    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322929    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322932    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322935    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322937    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322940    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322943    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322945    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322948    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322951    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322953    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.322987 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322956    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322959    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322961    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322966    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322970    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322979    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322982    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322984    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322987    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322990    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322993    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322996    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.322998    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323001    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323004    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323006    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323009    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323011    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323014    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323017    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.323486 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323019    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323022    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323024    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323027    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323029    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323032    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323034    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323038    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323040    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323044    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323046    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323049    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323051    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323055    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323058    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323060    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323063    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323066    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323069    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323072    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.323968 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323075    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323077    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323080    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323083    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323085    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323088    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323091    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323093    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323095    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323099    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323103    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323107    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323110    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323113    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323116    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323119    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323122    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323124    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323127    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.324454 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323130    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323133    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323135    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323138    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323140    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323144    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323147    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323151    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323153    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323156    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323159    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323161    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323164    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323166    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323169    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323171    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323174    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323176    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323179    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.324923 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323181    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323183    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323186    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323188    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323191    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.323193    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324599    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324608    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324613    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324617    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324620    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324623    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324626    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324629    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324631    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324635    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324638    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324641    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324643    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.325365 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324646    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324650    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324653    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324656    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324658    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324661    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324663    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324666    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324669    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324672    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324674    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324677    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324680    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324683    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324685    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324688    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324691    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324693    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324696    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324698    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.325832 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324701    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324703    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324706    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324708    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324711    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324715    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324718    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324720    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324723    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324726    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324728    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324731    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324733    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324736    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324739    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324741    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324745    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324747    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324750    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.326309 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324752    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324755    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324757    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324760    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324763    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324765    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324767    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324770    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324773    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324775    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324778    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324780    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324783    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324786    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324788    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324791    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324794    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324797    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324799    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324802    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.326809 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324805    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324807    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324810    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324812    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324815    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324817    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324820    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324823    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324825    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324828    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324831    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324833    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324836    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324838    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324841    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324843    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324846    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324848    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324851    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.327373 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324853    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324857    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324861    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324864    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324867    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324870    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324873    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324875    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324881    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324884    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324887    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324889    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324892    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324895    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.324898    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325371    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325377    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325380    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325383    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325385    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.327835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325388    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325391    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325394    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325396    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325399    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325402    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325404    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325407    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325409    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325412    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325415    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325417    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325420    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325423    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325425    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325428    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325430    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325433    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325435    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325438    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.328299 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325441    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325444    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325446    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325450    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325455    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325464    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325468    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325471    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325474    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325476    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325479    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325481    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325484    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325486    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325489    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325491    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325494    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325496    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325499    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.328778 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325501    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325504    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325506    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325509    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325511    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325514    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325518    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325521    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325525    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325543    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325547    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325550    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325552    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325555    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325558    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325564    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325567    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325570    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325572    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.329230 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325581    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325584    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325587    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325589    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325592    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325594    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325597    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325600    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325602    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325605    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325608    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325611    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325613    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325616    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325619    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325621    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325624    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325626    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325628    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325631    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.329680 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325633    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325636    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325640    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325643    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325645    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325648    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325650    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325653    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325657    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325660    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325662    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325665    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325667    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325671    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325673    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325676    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325678    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325682    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325684    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.330146 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325687    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325689    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325692    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.325694    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326455    2586 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326464    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326471    2586 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326476    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326480    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326484    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326488    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326492    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326496    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326499    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326503    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326506    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326509    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326512    2586 flags.go:64] FLAG: --cgroup-root=""
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326515    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326518    2586 flags.go:64] FLAG: --client-ca-file=""
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326522    2586 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326526    2586 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326541    2586 flags.go:64] FLAG: --cluster-domain=""
Nov 01 08:31:26.330590 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326545    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326548    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326551    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326554    2586 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326559    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326562    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326565    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326568    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326571    2586 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326574    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326577    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326580    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326583    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326588    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326590    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326593    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326596    2586 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326599    2586 flags.go:64] FLAG: --enable-server="true"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326602    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326607    2586 flags.go:64] FLAG: --event-burst="100"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326610    2586 flags.go:64] FLAG: --event-qps="50"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326613    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326616    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326620    2586 flags.go:64] FLAG: --eviction-hard=""
Nov 01 08:31:26.331124 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326625    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326628    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326631    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326634    2586 flags.go:64] FLAG: --eviction-soft=""
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326637    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326640    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326643    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326646    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326649    2586 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326652    2586 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326659    2586 flags.go:64] FLAG: --feature-gates=""
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326663    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326667    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326670    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326674    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326677    2586 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326680    2586 flags.go:64] FLAG: --help="false"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326683    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326686    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326689    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326692    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326695    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326699    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 08:31:26.331722 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326702    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326705    2586 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326708    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326711    2586 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326713    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326716    2586 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326720    2586 flags.go:64] FLAG: --kube-reserved=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326723    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326727    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326731    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326734    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326737    2586 flags.go:64] FLAG: --lock-file=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326739    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326742    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326746    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326751    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326754    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326757    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326760    2586 flags.go:64] FLAG: --logging-format="text"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326763    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326767    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326769    2586 flags.go:64] FLAG: --manifest-url=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326772    2586 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326778    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326781    2586 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 08:31:26.332272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326785    2586 flags.go:64] FLAG: --max-pods="110"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326788    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326791    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326794    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326797    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326800    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326803    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326806    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326813    2586 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326816    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326819    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326822    2586 flags.go:64] FLAG: --pod-cidr=""
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326826    2586 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326828    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326832    2586 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326836    2586 flags.go:64] FLAG: --port="10250"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326839    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326842    2586 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0acdb4ce1cd176a56"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326845    2586 flags.go:64] FLAG: --qos-reserved=""
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326848    2586 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326851    2586 flags.go:64] FLAG: --register-node="true"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326854    2586 flags.go:64] FLAG: --register-with-taints=""
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326858    2586 flags.go:64] FLAG: --registry-burst="10"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326861    2586 flags.go:64] FLAG: --registry-qps="5"
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326864    2586 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326867    2586 flags.go:64] FLAG: --reserved-memory=""
Nov 01 08:31:26.333075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326871    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326874    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326877    2586 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326880    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326883    2586 flags.go:64] FLAG: --runonce="false"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326893    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326903    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326907    2586 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326910    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326913    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326916    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326920    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326923    2586 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326926    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326929    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326932    2586 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326935    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326939    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326942    2586 flags.go:64] FLAG: --system-cgroups=""
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326944    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326950    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326954    2586 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326957    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326961    2586 flags.go:64] FLAG: --tls-min-version=""
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326964    2586 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 08:31:26.334187 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326967    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326970    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326973    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326977    2586 flags.go:64] FLAG: --v="2"
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326986    2586 flags.go:64] FLAG: --version="false"
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326990    2586 flags.go:64] FLAG: --vmodule=""
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326994    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.326997    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327134    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327138    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327141    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327144    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327147    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327150    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327153    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327156    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327159    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327161    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327164    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327167    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327169    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327171    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.335287 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327174    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327177    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327179    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327182    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327184    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327187    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327191    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327193    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327196    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327199    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327201    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327204    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327206    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327211    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327214    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327216    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327219    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327221    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327224    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327227    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.336258 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327230    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327232    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327235    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327238    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327241    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327243    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327246    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327249    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327251    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327254    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327256    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327259    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327261    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327264    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327267    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327269    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327272    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327275    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327280    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.337139 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327283    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327286    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327289    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327292    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327295    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327297    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327301    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327303    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327306    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327308    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327311    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327313    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327316    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327319    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327321    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327324    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327327    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327330    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327332    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.337734 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327335    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327338    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327340    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327343    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327345    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327348    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327350    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327353    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327357    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327361    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327364    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327366    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327371    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327374    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327376    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327379    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327382    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327384    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327387    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.338490 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327392    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327394    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327397    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327399    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327402    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327405    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327407    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327410    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327412    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327415    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327417    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327420    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327422    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327425    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327428    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.327466    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:31:26.339213 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327508    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327512    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327515    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327518    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327522    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327525    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327528    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327543    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327546    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327548    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327553    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327555    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327558    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327561    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327563    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327566    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327570    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327573    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327576    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.339842 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327580    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327583    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327585    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327588    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327590    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327593    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327596    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327598    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327601    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327603    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327606    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327609    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327612    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327614    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327617    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327620    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327622    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327625    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327627    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327629    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.340296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327632    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327635    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327637    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327641    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327644    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327647    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327650    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327652    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327655    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327659    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327662    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327665    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327669    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327671    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327674    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327676    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327679    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327682    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327684    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327687    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.340835 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327690    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327692    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327695    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327698    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327701    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327703    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327706    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327709    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327711    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327714    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327716    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327719    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327721    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327724    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327726    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327730    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327733    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327737    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327739    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.341296 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327742    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327744    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327748    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327751    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327753    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327756    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327758    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327761    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327764    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327766    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327769    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327772    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327774    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327777    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327779    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327782    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327784    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327788    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327792    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.341827 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327795    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327799    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327803    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327806    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327809    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327812    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327814    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327817    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.327819    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.335075    2586 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.335100    2586 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.335120    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.335127    2586 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335209    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335219    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335225    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335230    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335234    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335238    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.342412 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335243    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335247    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335252    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335257    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335261    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335265    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335270    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335275    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335279    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335283    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335287    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335291    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335295    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335299    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335303    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335307    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335311    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335315    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335320    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.343190 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335324    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335329    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335333    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335337    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335343    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335348    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335352    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335356    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335361    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335365    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335369    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335374    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335379    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335383    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335387    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335391    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335396    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335400    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335404    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.343981 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335408    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335413    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335417    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335421    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335426    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335430    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335434    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335439    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335443    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335447    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335451    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335455    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335460    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335467    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335473    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335477    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335482    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335486    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335491    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.344658 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335495    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335500    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335504    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335508    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335512    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335517    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335521    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335526    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335545    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335550    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335554    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335559    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335562    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335566    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335571    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335575    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335579    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335584    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335588    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335592    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.345129 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335596    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335600    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335605    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335609    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335613    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335617    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335622    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335626    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335630    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335635    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335639    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335644    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335649    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335653    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335657    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335661    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335666    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335671    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335675    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335679    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.345635 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335683    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335688    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335692    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.335733    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335945    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335952    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335956    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335961    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335965    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335969    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335974    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335979    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335983    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335987    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335991    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.335996    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.346094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336000    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336004    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336009    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336013    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336018    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336022    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336027    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336031    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336035    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336040    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336044    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336048    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336053    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336057    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336062    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336066    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336070    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336074    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336078    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336082    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.346611 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336087    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336091    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336095    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336099    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336103    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336107    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336112    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336116    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336122    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336128    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336133    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336137    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336142    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336146    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336151    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336155    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336160    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336164    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336168    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.347390 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336172    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336176    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336180    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336185    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336190    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336194    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336198    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336202    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336206    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336210    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336214    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336218    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336223    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336227    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336231    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336235    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336239    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336243    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336247    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.348060 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336251    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336255    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336259    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336263    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336266    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336271    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336276    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336280    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336284    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336289    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336293    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336298    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336302    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336307    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336311    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336315    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336320    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336325    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336329    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336333    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.348630 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336337    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336341    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336345    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336350    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336354    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336358    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336362    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336369    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336375    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336380    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336384    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336388    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336393    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336397    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336401    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336405    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.349094 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.336442    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336492    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336498    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336503    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336509    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336514    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336519    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336523    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336527    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336550    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336555    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336559    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336563    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336567    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336572    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336578    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336584    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336590    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336594    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:26.349480 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336599    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336603    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336607    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336611    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336615    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336619    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336624    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336628    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336633    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336637    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336641    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336645    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336650    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336654    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336659    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336663    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336667    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336671    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336675    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:26.350184 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336679    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336684    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336688    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336691    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336696    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336700    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336705    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336709    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336713    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336723    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336728    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336732    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336736    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336740    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336745    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336749    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336753    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336757    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336761    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336765    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:26.350652 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336769    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336773    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336777    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336781    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336788    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336793    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336799    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336805    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336810    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336815    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336819    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336825    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336829    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336833    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336838    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336842    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336846    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336850    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336854    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:26.351118 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336860    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336864    2586 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336868    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336873    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336877    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336881    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336885    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336889    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336894    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336898    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336902    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336906    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336910    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336914    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336919    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336923    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336927    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336932    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336937    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336941    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:26.351633 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336945    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336949    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336953    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336958    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336962    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336966    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336971    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336975    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336979    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:26.336984    2586 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.337064    2586 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.339638    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.340595    2586 server.go:1014] "Starting client certificate rotation"
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.340692    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.340728    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.342748    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:31:26.352095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.344456    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:31:26.356499 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.356484    2586 log.go:25] "Validated CRI v1 runtime API"
Nov 01 08:31:26.359358 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.359344    2586 log.go:25] "Validated CRI v1 image API"
Nov 01 08:31:26.361310 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.361290    2586 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 08:31:26.367954 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.367930    2586 fs.go:135] Filesystem UUIDs: map[05f5aa4e-a328-4f35-9380-895bf6cc8375:/dev/nvme0n1p3 62222a2b-8911-4ad7-a74d-f4a600405243:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 08:31:26.368014 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.367953    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 08:31:26.373194 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.373089    2586 manager.go:217] Machine: {Timestamp:2026-11-01 08:31:26.371736144 +0000 UTC m=+0.342785808 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103968 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f8d6e108aecf81e6138cf01320294 SystemUUID:ec2f8d6e-108a-ecf8-1e61-38cf01320294 BootID:3096594e-19aa-40f7-a9f7-03f6acf79c9e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ce:f8:59:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ce:f8:59:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:93:3e:30:2b:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 08:31:26.373194 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.373192    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 08:31:26.373337 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.373260    2586 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 08:31:26.376715 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.376690    2586 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 08:31:26.376841 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.376717    2586 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-93.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 08:31:26.376888 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.376844    2586 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 08:31:26.376888 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.376851    2586 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 08:31:26.376888 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.376885    2586 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:31:26.376968 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.376904    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:31:26.377614 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.377604    2586 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 08:31:26.381524 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.381511    2586 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 08:31:26.381685 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.381676    2586 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 08:31:26.382466 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.382456    2586 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 08:31:26.382512 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.382471    2586 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 08:31:26.382512 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.382489    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 08:31:26.382512 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.382505    2586 kubelet.go:405] "Adding apiserver pod source"
Nov 01 08:31:26.382657 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.382522    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 08:31:26.382822 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.382809    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:31:26.385064 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.385045    2586 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 08:31:26.390632 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.390619    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:31:26.391482 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391470    2586 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 08:31:26.391516 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391501    2586 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 08:31:26.391696 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391685    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 08:31:26.391725 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391710    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 08:31:26.391725 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391724    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 08:31:26.391782 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391736    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 08:31:26.391782 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391745    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 08:31:26.391782 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.391753    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 08:31:26.392350 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392342    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 08:31:26.392377 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392351    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 08:31:26.392377 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392359    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 08:31:26.392377 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392365    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 08:31:26.392377 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392374    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 08:31:26.392555 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392382    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 08:31:26.392878 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392869    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 08:31:26.392910 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.392879    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 08:31:26.396353 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.396342    2586 server.go:1263] "Started kubelet"
Nov 01 08:31:26.396564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.396428    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 08:31:26.396630 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.396605    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 08:31:26.396681 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.396547    2586 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 08:31:26.397055 ip-10-0-134-93 systemd[1]: Started Kubernetes Kubelet.
Nov 01 08:31:26.397556 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.397506    2586 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 08:31:26.404246 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.404224    2586 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 08:31:26.406721 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.406704    2586 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 08:31:26.406805 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.406735    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 08:31:26.406871 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.406821    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:26.407336 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407312    2586 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 08:31:26.407336 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407335    2586 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 08:31:26.407459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407370    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 08:31:26.407495 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407453    2586 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 08:31:26.407495 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407468    2586 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 08:31:26.407608 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407580    2586 factory.go:55] Registering systemd factory
Nov 01 08:31:26.407664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407614    2586 factory.go:223] Registration of the systemd container factory successfully
Nov 01 08:31:26.407914 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407889    2586 factory.go:153] Registering CRI-O factory
Nov 01 08:31:26.407914 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407904    2586 factory.go:223] Registration of the crio container factory successfully
Nov 01 08:31:26.408034 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407951    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 08:31:26.408034 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407968    2586 factory.go:103] Registering Raw factory
Nov 01 08:31:26.408034 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.407966    2586 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 08:31:26.408034 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.407981    2586 manager.go:1199] Started watching for new ooms in manager
Nov 01 08:31:26.408787 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.408774    2586 manager.go:319] Starting recovery of all containers
Nov 01 08:31:26.422569 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.422552    2586 manager.go:324] Recovery completed
Nov 01 08:31:26.426978 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.426964    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.429107 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.429069    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.429202 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.429127    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.429202 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.429143    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.429832 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.429816    2586 cpu_manager.go:226] "Starting" policy="none"
Nov 01 08:31:26.429832 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.429829    2586 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 08:31:26.429957 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.429850    2586 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 08:31:26.430038 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.430023    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.431282 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.431263    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.431373 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.431295    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.431373 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.431309    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.432156 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.432142    2586 policy_none.go:50] "Start"
Nov 01 08:31:26.432205 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.432169    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 08:31:26.432205 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.432185    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 08:31:26.434264 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.434252    2586 policy_none.go:44] "Start"
Nov 01 08:31:26.489439 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.477471    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 08:31:26.489439 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.478521    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 08:31:26.489439 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.478565    2586 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 08:31:26.489439 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.478588    2586 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 08:31:26.489439 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.478628    2586 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 08:31:26.507922 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.507901    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:26.536044 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.536020    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-93.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 08:31:26.537021 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.536049    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-93.us-west-2.compute.internal.18e3dea7378a7ca6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-93.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-93.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:26.396316838 +0000 UTC m=+0.367366501,LastTimestamp:2026-11-01 08:31:26.396316838 +0000 UTC m=+0.367366501,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-93.us-west-2.compute.internal,}"
Nov 01 08:31:26.579498 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.579484    2586 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 08:31:26.581832 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.581800    2586 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-93.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 08:31:26.599948 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.599881    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-93.us-west-2.compute.internal.18e3dea7397eeb6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-93.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-93.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-93.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:26.429113195 +0000 UTC m=+0.400162870,LastTimestamp:2026-11-01 08:31:26.429113195 +0000 UTC m=+0.400162870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-93.us-west-2.compute.internal,}"
Nov 01 08:31:26.600242 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.600223    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 08:31:26.600347 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.600328    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-93.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 08:31:26.600437 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.600418    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 08:31:26.603081 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.603054    2586 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 08:31:26.605273 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605260    2586 manager.go:341] "Starting Device Plugin manager"
Nov 01 08:31:26.605370 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.605334    2586 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 08:31:26.605370 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605347    2586 server.go:87] "Starting device plugin registration server"
Nov 01 08:31:26.605628 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605615    2586 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 08:31:26.605697 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605629    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 08:31:26.605760 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605744    2586 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 08:31:26.605837 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605825    2586 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 08:31:26.605878 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.605838    2586 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 08:31:26.606425 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.606403    2586 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 08:31:26.606511 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.606454    2586 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:26.627098 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.627074    2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 08:31:26.627174 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.627156    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9sh9w"
Nov 01 08:31:26.696240 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.696214    2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:26.705792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.705770    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.706864 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.706842    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.706938 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.706881    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.706938 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.706909    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.706938 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.706933    2586 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.764827 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.764804    2586 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.764904 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.764829    2586 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-93.us-west-2.compute.internal\": node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:26.781611 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.781590    2586 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"]
Nov 01 08:31:26.781661 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.781647    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.782630 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.782613    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.782707 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.782640    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.782707 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.782652    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.784474 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.784461    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.784654 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.784641    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.784698 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.784668    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.785299 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.785283    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.785362 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.785307    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.785362 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.785317    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.785428 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.785283    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.785428 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.785393    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.785428 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.785405    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.787116 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.787096    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.787116 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.787120    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:26.787778 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.787765    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:26.787857 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.787790    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:26.787857 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.787804    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:26.803359 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.803344    2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.806198 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.806181    2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.809673 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.809657    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.809719 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.809682    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.809719 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.809699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67982dcee5ed5a322620afe889b9e391-config\") pod \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"67982dcee5ed5a322620afe889b9e391\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.877352 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.877291    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:26.910353 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.910327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.910470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.910358    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.910470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.910378    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67982dcee5ed5a322620afe889b9e391-config\") pod \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"67982dcee5ed5a322620afe889b9e391\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.910470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.910440    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.910470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.910451    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.910626 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:26.910440    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67982dcee5ed5a322620afe889b9e391-config\") pod \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"67982dcee5ed5a322620afe889b9e391\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:26.978205 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:26.978182    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.079287 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.079253    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.105981 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.105963    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:27.108871 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.108847    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:27.179595 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.179339    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.281351 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.281310    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.341473 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.341433    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 08:31:27.342014 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.341652    2586 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 08:31:27.342014 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.341642    2586 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-93.us-west-2.compute.internal.18e3dea7397eeb6b\": read tcp 10.0.134.93:45490->3.130.67.7:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-93.us-west-2.compute.internal.18e3dea7397eeb6b  default   4808 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-93.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-93.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-93.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:26 +0000 UTC,LastTimestamp:2026-11-01 08:31:26.782628289 +0000 UTC m=+0.753677952,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-93.us-west-2.compute.internal,}"
Nov 01 08:31:27.382414 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.382385    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.482710 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.482631    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.551952 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.551928    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:27.553011 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.552990    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:27.553130 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.553027    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:27.553130 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.553043    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:27.580877 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.580844    2586 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:27.583309 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.583295    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.630910 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.630871    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 08:26:27 +0000 UTC" deadline="2028-04-22 13:06:09.900701968 +0000 UTC"
Nov 01 08:31:27.630910 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.630906    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12916h34m42.269799406s"
Nov 01 08:31:27.682485 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:27.682457    2586 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:27.683444 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.683426    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.786883 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.786816    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.886945 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.886907    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:27.987019 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:27.986987    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:28.087395 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.087311    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:31:28.142988 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.142958    2586 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:28.207705 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.207673    2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:28.258748 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.258714    2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:28.270095 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.270073    2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:31:28.271109 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.271095    2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:31:28.311435 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.311406    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797457622f1c6f5ba3270c9efea90468.slice/crio-020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63 WatchSource:0}: Error finding container 020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63: Status 404 returned error can't find the container with id 020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63
Nov 01 08:31:28.319710 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.319691    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:31:28.322793 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.322770    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67982dcee5ed5a322620afe889b9e391.slice/crio-b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62 WatchSource:0}: Error finding container b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62: Status 404 returned error can't find the container with id b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62
Nov 01 08:31:28.330688 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.330658    2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:31:28.384011 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.383931    2586 apiserver.go:52] "Watching apiserver"
Nov 01 08:31:28.408641 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.408610    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 08:31:28.483773 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.483718    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"67982dcee5ed5a322620afe889b9e391","Type":"ContainerStarted","Data":"b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62"}
Nov 01 08:31:28.484809 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.484778    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"797457622f1c6f5ba3270c9efea90468","Type":"ContainerStarted","Data":"020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63"}
Nov 01 08:31:28.492374 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.492350    2586 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 08:31:28.492782 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.492756    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-92bsb","openshift-multus/network-metrics-daemon-rkhtc","openshift-cluster-node-tuning-operator/tuned-nw2n6","openshift-dns/node-resolver-sjnbq","openshift-multus/multus-additional-cni-plugins-8cngb","openshift-network-operator/iptables-alerter-bgtrj","kube-system/global-pull-secret-syncer-5cgcj","openshift-image-registry/node-ca-p6sk5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-4lxtb","openshift-ovn-kubernetes/ovnkube-node-m9vpp","kube-system/konnectivity-agent-slw5x","kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"]
Nov 01 08:31:28.496697 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.496677    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:28.496800 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.496741    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:28.499099 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.499082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.501575 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.501557    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.503953 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.503935    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.506338 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.506321    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.508880 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.508859    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.511496 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.511471    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.517001 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.514234    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:28.517001 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.514347    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:28.518519 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.518497    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.518655 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.518605    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:28.520912 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.520896    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.522980 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.522955    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0369242-85ce-442a-a759-cd0d52c7fdef-iptables-alerter-script\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.523082 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.522995    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-modprobe-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.523082 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523024    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-systemd-units\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523082 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523049    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-netns\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523082 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523069    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-systemd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-etc-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523134    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b827d2-1aac-4471-9665-21c2575f6d0c-tmp-dir\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523149    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0369242-85ce-442a-a759-cd0d52c7fdef-host-slash\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523165    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-kubelet\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-slash\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523206    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-ovn\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523221    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-script-lib\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523262 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523245    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-systemd\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523267    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-sys\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cqz\" (UniqueName: \"kubernetes.io/projected/c9b827d2-1aac-4471-9665-21c2575f6d0c-kube-api-access-j6cqz\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523335    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523341    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523441    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-cnibin\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.523557 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523574    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523612    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfq5\" (UniqueName: \"kubernetes.io/projected/a7c7f909-3dd3-473a-aacd-356bce893dcb-kube-api-access-ztfq5\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523643    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-host\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjbx\" (UniqueName: \"kubernetes.io/projected/e0369242-85ce-442a-a759-cd0d52c7fdef-kube-api-access-jmjbx\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-env-overrides\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523776    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysconfig\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-netd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523846    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523865    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9b827d2-1aac-4471-9665-21c2575f6d0c-hosts-file\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523896    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bsq\" (UniqueName: \"kubernetes.io/projected/65196622-05c5-4993-be70-8921dfd83293-kube-api-access-n2bsq\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523925    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.523967 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523956    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-node-log\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.523983    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-log-socket\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524007    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-bin\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524033    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-config\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-lib-modules\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524076    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-tuned\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-tmp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524140    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-var-lib-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524176    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-kubernetes\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524232    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-conf\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524246    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-var-lib-kubelet\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524284    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqvp\" (UniqueName: \"kubernetes.io/projected/083acc55-e6ba-4016-abad-67be0cc4af2c-kube-api-access-bwqvp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524310    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-run\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524327    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-system-cni-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.524459 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.524343    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.525889 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.525869    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.552404 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.552381    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 08:31:28.552766 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.552741    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 08:31:28.552766 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.552763    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cdn26\""
Nov 01 08:31:28.552997 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.552743    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.552997 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.552774    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.552997 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.552783    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 08:31:28.553131 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.553045    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 08:31:28.554461 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.554439    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.554553 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.554464    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.554553 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.554512    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48mlk\""
Nov 01 08:31:28.556971 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.556950    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.557064 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.557027    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.557133 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.557103    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d96l\""
Nov 01 08:31:28.559693 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.559661    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 08:31:28.559789 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.559673    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.559789 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.559781    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 08:31:28.559905 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.559844    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hnxt\""
Nov 01 08:31:28.559905 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.559862    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.559979 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.559928    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 08:31:28.561943 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.561927    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.562098 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.562083    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xk5lv\""
Nov 01 08:31:28.562339 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.562327    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 08:31:28.562835 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.562820    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.564521 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.564505    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 08:31:28.564592 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.564544    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9vpz\""
Nov 01 08:31:28.578318 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.578302    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 08:31:28.595876 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.595849    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 08:31:28.596408 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596384    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.596564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596441    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.596564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596460    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 08:31:28.596564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596462    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 08:31:28.596564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596500    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 08:31:28.596564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596527    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 08:31:28.596754 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596555    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24cps\""
Nov 01 08:31:28.596754 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596637    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxggb\""
Nov 01 08:31:28.596754 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.596647    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqp75\""
Nov 01 08:31:28.599201 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.599183    2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 08:31:28.599279 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.599212    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmcqn"
Nov 01 08:31:28.608865 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.608844    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 08:31:28.625624 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625599    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-socket-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.625717 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625627    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-device-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.625717 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625655    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.625786 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfq5\" (UniqueName: \"kubernetes.io/projected/a7c7f909-3dd3-473a-aacd-356bce893dcb-kube-api-access-ztfq5\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.625786 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-host\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.625786 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625769    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-socket-dir-parent\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.625883 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625786    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bade5291-3729-42d3-92da-7fe9e224a4ad-konnectivity-ca\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.625883 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625783    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.625883 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.625883 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625870    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-host\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626004 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.625919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysconfig\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626022    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9b827d2-1aac-4471-9665-21c2575f6d0c-hosts-file\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.626070 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626049    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysconfig\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626104 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bsq\" (UniqueName: \"kubernetes.io/projected/65196622-05c5-4993-be70-8921dfd83293-kube-api-access-n2bsq\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.626149 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626102    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-hostroot\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.626149 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626125    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-daemon-config\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.626256 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626141    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9b827d2-1aac-4471-9665-21c2575f6d0c-hosts-file\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.626256 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626185    2586 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 08:31:28.626256 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626234    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.626363 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626265    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bade5291-3729-42d3-92da-7fe9e224a4ad-agent-certs\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.626363 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626293    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.626363 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.626363 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626338    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-dbus\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.626502 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626363    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-var-lib-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.626502 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626385    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-kubernetes\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626502 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626430    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626502 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626440    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-kubernetes\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626502 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-conf\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626746 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626532    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-var-lib-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.626746 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626570    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626746 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-var-lib-kubelet\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626746 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626612    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-var-lib-kubelet\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.626746 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626631    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-conf\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.627075 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.626624    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ad972a-7efb-4755-bdae-e2087f130fb4-host\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-run\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs477\" (UniqueName: \"kubernetes.io/projected/55eaf339-45f2-4182-bcab-a01f489180ee-kube-api-access-hs477\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627625    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-run\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627649    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-modprobe-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627711    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-systemd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627747    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-etc-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627780    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0369242-85ce-442a-a759-cd0d52c7fdef-host-slash\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627826    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-multus-certs\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627845    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-modprobe-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.627848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627854    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62767\" (UniqueName: \"kubernetes.io/projected/47ad972a-7efb-4755-bdae-e2087f130fb4-kube-api-access-62767\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627888    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0369242-85ce-442a-a759-cd0d52c7fdef-host-slash\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627895    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-etc-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627932    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-slash\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627943    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-systemd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.627983    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-sys\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cqz\" (UniqueName: \"kubernetes.io/projected/c9b827d2-1aac-4471-9665-21c2575f6d0c-kube-api-access-j6cqz\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628051    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-k8s-cni-cncf-io\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628068    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-sys\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628074    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-kubelet\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628131    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-cnibin\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628203    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjbx\" (UniqueName: \"kubernetes.io/projected/e0369242-85ce-442a-a759-cd0d52c7fdef-kube-api-access-jmjbx\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628226    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-registration-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628246    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-sys-fs\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628273    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-cnibin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.628445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-bin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628447    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-conf-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628480    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-env-overrides\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628506    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-netd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6czd\" (UniqueName: \"kubernetes.io/projected/c3286913-2025-4768-8e0f-65467b292ccc-kube-api-access-s6czd\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628810    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-cnibin\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.628845    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-slash\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629141    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629217    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629269 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629265    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-netd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629308    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-system-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629336    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-kubelet-config\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629422    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-node-log\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629451    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-log-socket\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629483    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-bin\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629561    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-config\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629566    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-env-overrides\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629641    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-lib-modules\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629653    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-node-log\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629680    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-tuned\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.629792 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629767    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629809    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-log-socket\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629896    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-lib-modules\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629934    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-tmp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.629969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqvp\" (UniqueName: \"kubernetes.io/projected/083acc55-e6ba-4016-abad-67be0cc4af2c-kube-api-access-bwqvp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630057    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630087    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-cni-binary-copy\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-etc-kubernetes\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630145    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-config\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-system-cni-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630212    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-bin\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630291    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0369242-85ce-442a-a759-cd0d52c7fdef-iptables-alerter-script\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-netns\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.630383 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630365    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-systemd-units\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630676    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-netns\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630715    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b827d2-1aac-4471-9665-21c2575f6d0c-tmp-dir\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630781    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-multus\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630815    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ad972a-7efb-4755-bdae-e2087f130fb4-serviceca\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-kubelet\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630881    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-ovn\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-script-lib\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630953    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-systemd\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.630987    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.631037 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.631500 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631158    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5fr\" (UniqueName: \"kubernetes.io/projected/3c33083a-5961-4854-a5af-8b1de96e6eec-kube-api-access-8b5fr\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.631500 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631336    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-systemd-units\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631601 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631496    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631601 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631534    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-system-cni-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.631601 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631620    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-ovn\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631784 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631641    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-kubelet\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631784 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631699    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-systemd\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.631892 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631857    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-netns\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.631959 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.631921    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.632090 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.632057    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b827d2-1aac-4471-9665-21c2575f6d0c-tmp-dir\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.632237 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.632182    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0369242-85ce-442a-a759-cd0d52c7fdef-iptables-alerter-script\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.632237 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.632187    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.633110 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.633088    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-script-lib\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.633270 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.633250    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.633824 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.633803    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-tuned\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.633894 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.633830    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-tmp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.688877 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.688793    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:28.688877 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.688830    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:28.688877 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.688847    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:28.689128 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.688939    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:29.188912378 +0000 UTC m=+3.159962054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:28.689568 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.689546    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfq5\" (UniqueName: \"kubernetes.io/projected/a7c7f909-3dd3-473a-aacd-356bce893dcb-kube-api-access-ztfq5\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.689647 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.689619    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bsq\" (UniqueName: \"kubernetes.io/projected/65196622-05c5-4993-be70-8921dfd83293-kube-api-access-n2bsq\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.690365 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.690342    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cqz\" (UniqueName: \"kubernetes.io/projected/c9b827d2-1aac-4471-9665-21c2575f6d0c-kube-api-access-j6cqz\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.690481 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.690462    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjbx\" (UniqueName: \"kubernetes.io/projected/e0369242-85ce-442a-a759-cd0d52c7fdef-kube-api-access-jmjbx\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.715148 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.715124    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqvp\" (UniqueName: \"kubernetes.io/projected/083acc55-e6ba-4016-abad-67be0cc4af2c-kube-api-access-bwqvp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.731623 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-k8s-cni-cncf-io\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.731704 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731630    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-kubelet\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.731704 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731663    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-registration-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.731771 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731728    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-kubelet\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.731771 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731716    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-k8s-cni-cncf-io\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.731771 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731734    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-registration-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.731865 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731785    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-sys-fs\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.731865 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731814    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-cnibin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.731865 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731833    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-bin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.731865 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731849    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-conf-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731847    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-sys-fs\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6czd\" (UniqueName: \"kubernetes.io/projected/c3286913-2025-4768-8e0f-65467b292ccc-kube-api-access-s6czd\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731893    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-cnibin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731909    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-system-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-kubelet-config\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731880    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-conf-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731957    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731984    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-system-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731990    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-cni-binary-copy\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-etc-kubernetes\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.732029    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732030    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-netns\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732065    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-netns\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.731882    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-bin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.732087    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:31:29.232068455 +0000 UTC m=+3.203118122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732094    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-kubelet-config\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732132    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-etc-kubernetes\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732152    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-multus\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ad972a-7efb-4755-bdae-e2087f130fb4-serviceca\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5fr\" (UniqueName: \"kubernetes.io/projected/3c33083a-5961-4854-a5af-8b1de96e6eec-kube-api-access-8b5fr\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732217    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-socket-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732215    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-multus\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-device-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732259    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-socket-dir-parent\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732275    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bade5291-3729-42d3-92da-7fe9e224a4ad-konnectivity-ca\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732293    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-hostroot\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.732424 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732315    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-daemon-config\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732342    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-socket-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732358    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-socket-dir-parent\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bade5291-3729-42d3-92da-7fe9e224a4ad-agent-certs\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732389    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-hostroot\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.732406    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732416    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732430    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-device-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732450    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:28.732467    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret podName:3370a9e2-dba0-475a-a84e-3d0d1cfab280 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:29.232448555 +0000 UTC m=+3.203498222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret") pod "global-pull-secret-syncer-5cgcj" (UID: "3370a9e2-dba0-475a-a84e-3d0d1cfab280") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732483    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732502    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-dbus\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732538    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ad972a-7efb-4755-bdae-e2087f130fb4-host\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732577    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732587    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs477\" (UniqueName: \"kubernetes.io/projected/55eaf339-45f2-4182-bcab-a01f489180ee-kube-api-access-hs477\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732615    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733138 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732623    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-dbus\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732645    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-multus-certs\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732671    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62767\" (UniqueName: \"kubernetes.io/projected/47ad972a-7efb-4755-bdae-e2087f130fb4-kube-api-access-62767\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732695    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732626    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ad972a-7efb-4755-bdae-e2087f130fb4-host\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732711    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ad972a-7efb-4755-bdae-e2087f130fb4-serviceca\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732753    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-multus-certs\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732760    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-cni-binary-copy\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.732925    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-daemon-config\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.733664 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.733265    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bade5291-3729-42d3-92da-7fe9e224a4ad-konnectivity-ca\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.734977 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.734956    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bade5291-3729-42d3-92da-7fe9e224a4ad-agent-certs\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.794748 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.794716    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62767\" (UniqueName: \"kubernetes.io/projected/47ad972a-7efb-4755-bdae-e2087f130fb4-kube-api-access-62767\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.794748 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.794726    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5fr\" (UniqueName: \"kubernetes.io/projected/3c33083a-5961-4854-a5af-8b1de96e6eec-kube-api-access-8b5fr\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.794748 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.794743    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs477\" (UniqueName: \"kubernetes.io/projected/55eaf339-45f2-4182-bcab-a01f489180ee-kube-api-access-hs477\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.794931 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.794829    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6czd\" (UniqueName: \"kubernetes.io/projected/c3286913-2025-4768-8e0f-65467b292ccc-kube-api-access-s6czd\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:28.823126 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.823098    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:31:28.829364 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.829341    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:28.833245 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.833224    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:31:28.833640 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.833623    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:31:28.833704 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.833650    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083acc55_e6ba_4016_abad_67be0cc4af2c.slice/crio-ec263a5d533bc158852b29b4106d35b37b544cb88894aca38b9f73beb8b0961e WatchSource:0}: Error finding container ec263a5d533bc158852b29b4106d35b37b544cb88894aca38b9f73beb8b0961e: Status 404 returned error can't find the container with id ec263a5d533bc158852b29b4106d35b37b544cb88894aca38b9f73beb8b0961e
Nov 01 08:31:28.838695 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.838671    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c7f909_3dd3_473a_aacd_356bce893dcb.slice/crio-2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28 WatchSource:0}: Error finding container 2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28: Status 404 returned error can't find the container with id 2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28
Nov 01 08:31:28.843347 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.843329    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0369242_85ce_442a_a759_cd0d52c7fdef.slice/crio-8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1 WatchSource:0}: Error finding container 8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1: Status 404 returned error can't find the container with id 8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1
Nov 01 08:31:28.846158 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.846078    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b827d2_1aac_4471_9665_21c2575f6d0c.slice/crio-8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0 WatchSource:0}: Error finding container 8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0: Status 404 returned error can't find the container with id 8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0
Nov 01 08:31:28.851322 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.851296    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-92bsb"
Nov 01 08:31:28.853592 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.853573    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:28.857552 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.857519    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:31:28.860290 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.860262    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:31:28.862138 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.862117    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c33083a_5961_4854_a5af_8b1de96e6eec.slice/crio-c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a WatchSource:0}: Error finding container c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a: Status 404 returned error can't find the container with id c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a
Nov 01 08:31:28.865994 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.865965    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbade5291_3729_42d3_92da_7fe9e224a4ad.slice/crio-769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd WatchSource:0}: Error finding container 769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd: Status 404 returned error can't find the container with id 769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd
Nov 01 08:31:28.867743 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:28.867723    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:31:28.871017 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.870998    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65196622_05c5_4993_be70_8921dfd83293.slice/crio-2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9 WatchSource:0}: Error finding container 2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9: Status 404 returned error can't find the container with id 2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9
Nov 01 08:31:28.871393 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.871337    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ad972a_7efb_4755_bdae_e2087f130fb4.slice/crio-4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb WatchSource:0}: Error finding container 4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb: Status 404 returned error can't find the container with id 4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb
Nov 01 08:31:28.878052 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:28.878034    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55eaf339_45f2_4182_bcab_a01f489180ee.slice/crio-039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad WatchSource:0}: Error finding container 039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad: Status 404 returned error can't find the container with id 039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad
Nov 01 08:31:29.236003 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.235971    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:29.236158 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.236014    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:29.236158 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.236060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:29.236158 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236111    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:29.236158 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236151    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:29.236289 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236162    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:31:30.236148033 +0000 UTC m=+4.207197710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:29.236289 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236167    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:29.236289 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236178    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:29.236289 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236202    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:29.236289 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236223    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:30.236207207 +0000 UTC m=+4.207256874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:29.236289 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:29.236239    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret podName:3370a9e2-dba0-475a-a84e-3d0d1cfab280 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:30.236232336 +0000 UTC m=+4.207281988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret") pod "global-pull-secret-syncer-5cgcj" (UID: "3370a9e2-dba0-475a-a84e-3d0d1cfab280") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:29.488743 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.488647    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nw2n6" event={"ID":"083acc55-e6ba-4016-abad-67be0cc4af2c","Type":"ContainerStarted","Data":"ec263a5d533bc158852b29b4106d35b37b544cb88894aca38b9f73beb8b0961e"}
Nov 01 08:31:29.490042 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.490016    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg" event={"ID":"55eaf339-45f2-4182-bcab-a01f489180ee","Type":"ContainerStarted","Data":"039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad"}
Nov 01 08:31:29.491314 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.491287    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6sk5" event={"ID":"47ad972a-7efb-4755-bdae-e2087f130fb4","Type":"ContainerStarted","Data":"4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb"}
Nov 01 08:31:29.492632 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.492607    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-slw5x" event={"ID":"bade5291-3729-42d3-92da-7fe9e224a4ad","Type":"ContainerStarted","Data":"769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd"}
Nov 01 08:31:29.493864 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.493830    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerStarted","Data":"2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9"}
Nov 01 08:31:29.495123 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.495103    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-92bsb" event={"ID":"3c33083a-5961-4854-a5af-8b1de96e6eec","Type":"ContainerStarted","Data":"c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a"}
Nov 01 08:31:29.496253 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.496232    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sjnbq" event={"ID":"c9b827d2-1aac-4471-9665-21c2575f6d0c","Type":"ContainerStarted","Data":"8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0"}
Nov 01 08:31:29.497446 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.497424    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bgtrj" event={"ID":"e0369242-85ce-442a-a759-cd0d52c7fdef","Type":"ContainerStarted","Data":"8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1"}
Nov 01 08:31:29.498764 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.498742    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28"}
Nov 01 08:31:29.603175 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.603127    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:29 +0000 UTC" deadline="2028-05-07 07:21:47.111991908 +0000 UTC"
Nov 01 08:31:29.603175 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:29.603169    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13270h50m17.508828703s"
Nov 01 08:31:30.248071 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.248021    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:30.248260 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.248099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:30.248260 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.248154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:30.248381 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248288    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:30.248381 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248349    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret podName:3370a9e2-dba0-475a-a84e-3d0d1cfab280 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:32.248332073 +0000 UTC m=+6.219381741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret") pod "global-pull-secret-syncer-5cgcj" (UID: "3370a9e2-dba0-475a-a84e-3d0d1cfab280") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:30.248492 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248440    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:30.248492 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248455    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:30.248492 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248468    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:30.248671 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248532    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:32.248493315 +0000 UTC m=+6.219542982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:30.248743 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248712    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:30.248802 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.248754    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:31:32.24874131 +0000 UTC m=+6.219790978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:30.482802 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.481928    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:30.482802 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.482078    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:30.482802 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.482482    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:30.482802 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.482576    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:30.482802 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.482665    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:30.482802 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:30.482742    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:30.603786 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.603694    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:29 +0000 UTC" deadline="2028-04-01 17:14:57.298470395 +0000 UTC"
Nov 01 08:31:30.603786 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:30.603758    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12416h43m26.694716966s"
Nov 01 08:31:32.268516 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:32.268407    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:32.268516 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:32.268469    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:32.268520    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268665    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268727    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret podName:3370a9e2-dba0-475a-a84e-3d0d1cfab280 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:36.268710263 +0000 UTC m=+10.239759933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret") pod "global-pull-secret-syncer-5cgcj" (UID: "3370a9e2-dba0-475a-a84e-3d0d1cfab280") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268816    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268835    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268848    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268885    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:36.268873469 +0000 UTC m=+10.239923136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268949    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:32.268996 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.268983    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:31:36.268970443 +0000 UTC m=+10.240020111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:32.481942 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:32.481908    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:32.482133 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.482037    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:32.482133 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:32.482095    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:32.482251 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.482161    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:32.482251 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:32.482183    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:32.482251 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:32.482244    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:33.536356 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:33.536259    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"67982dcee5ed5a322620afe889b9e391","Type":"ContainerStarted","Data":"443152204e948875d2b0390f5afdfb422f75866d3238b42edfd2c3646710383b"}
Nov 01 08:31:33.539684 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:33.539652    2586 generic.go:356] "Generic (PLEG): container finished" podID="797457622f1c6f5ba3270c9efea90468" containerID="5b52598feee215ca54e9e7379a81a41dc2b6ddb51d9f07cd95f6c439e055783f" exitCode=0
Nov 01 08:31:33.539813 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:33.539741    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"797457622f1c6f5ba3270c9efea90468","Type":"ContainerDied","Data":"5b52598feee215ca54e9e7379a81a41dc2b6ddb51d9f07cd95f6c439e055783f"}
Nov 01 08:31:33.551183 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:33.551147    2586 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="28f42d501d5cfbfa6c70ebf5699ab5cda45061812ec112079dc19f21ecdc7529" exitCode=0
Nov 01 08:31:33.551284 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:33.551199    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerDied","Data":"28f42d501d5cfbfa6c70ebf5699ab5cda45061812ec112079dc19f21ecdc7529"}
Nov 01 08:31:33.776215 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:33.776147    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal" podStartSLOduration=4.7761258909999995 podStartE2EDuration="4.776125891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:33.654760736 +0000 UTC m=+7.625810414" watchObservedRunningTime="2026-11-01 08:31:33.776125891 +0000 UTC m=+7.747175564"
Nov 01 08:31:34.481832 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:34.481805    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:34.481957 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:34.481938    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:34.482375 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:34.482354    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:34.482483 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:34.482461    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:34.482561 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:34.482535    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:34.482666 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:34.482646    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:34.560418 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:34.559745    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"797457622f1c6f5ba3270c9efea90468","Type":"ContainerStarted","Data":"4c22f652223c33f9f3f93966216b12a7b677e3548143724d171747a023caf838"}
Nov 01 08:31:34.564719 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:34.564269    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-slw5x" event={"ID":"bade5291-3729-42d3-92da-7fe9e224a4ad","Type":"ContainerStarted","Data":"8effc443197a1262501e7407562f4f4c2c45f6809db023fa0786240da3eec309"}
Nov 01 08:31:34.678584 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:34.678305    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" podStartSLOduration=5.678284777 podStartE2EDuration="5.678284777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:34.67750179 +0000 UTC m=+8.648551471" watchObservedRunningTime="2026-11-01 08:31:34.678284777 +0000 UTC m=+8.649334453"
Nov 01 08:31:35.573600 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:35.573553    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6sk5" event={"ID":"47ad972a-7efb-4755-bdae-e2087f130fb4","Type":"ContainerStarted","Data":"e297502e4f545278e8429756735521a5a1c37c070fd52f8d9d601de74ec10560"}
Nov 01 08:31:35.691564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:35.690437    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-slw5x" podStartSLOduration=2.228243006 podStartE2EDuration="7.690411809s" totalImagesPullingTime="5.462168803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:34.79585545 +0000 UTC m=+8.766905127" watchObservedRunningTime="2026-11-01 08:31:35.690411809 +0000 UTC m=+9.661461487"
Nov 01 08:31:36.305066 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.305024    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:36.305184 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.305092    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:36.305184 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.305136    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:36.305306 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305268    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:36.305363 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305327    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:31:44.305308209 +0000 UTC m=+18.276357876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:36.305430 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305393    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:36.305493 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305429    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret podName:3370a9e2-dba0-475a-a84e-3d0d1cfab280 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:44.305416982 +0000 UTC m=+18.276466654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret") pod "global-pull-secret-syncer-5cgcj" (UID: "3370a9e2-dba0-475a-a84e-3d0d1cfab280") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:36.305553 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305525    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:36.305553 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305545    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:36.305673 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305558    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:36.305673 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.305610    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:44.305582011 +0000 UTC m=+18.276631678 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:36.480909 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.480874    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:36.481094 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.480989    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:36.481094 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.481070    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:36.481217 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.481144    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:36.481389 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.481356    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:36.481552 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:36.481471    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:36.577723 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:36.577680    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg" event={"ID":"55eaf339-45f2-4182-bcab-a01f489180ee","Type":"ContainerStarted","Data":"62d302c56af7386a85b3ce9872d4245dcf3439290e356b3472f2bf4cd21fd285"}
Nov 01 08:31:38.479660 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.479061    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:38.479660 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.479078    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:38.479660 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.479072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:38.479660 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:38.479262    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:38.479660 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:38.479402    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:38.479660 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:38.479486    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:38.488369 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.488310    2586 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 08:31:38.584231 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.584191    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg" event={"ID":"55eaf339-45f2-4182-bcab-a01f489180ee","Type":"ContainerStarted","Data":"ebaac71797322672bc7a3a3f396ba0338b1a9a35fa39ad6e104d3bd6c10a84c6"}
Nov 01 08:31:38.586223 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.586195    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-92bsb" event={"ID":"3c33083a-5961-4854-a5af-8b1de96e6eec","Type":"ContainerStarted","Data":"3183f37a0d02def5fe6e9b812b89cce42f50888a8538ce635af41d57901d8040"}
Nov 01 08:31:38.588212 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.588146    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sjnbq" event={"ID":"c9b827d2-1aac-4471-9665-21c2575f6d0c","Type":"ContainerStarted","Data":"ac88f98e1a2144fb710c57a2cd8a1e3d816c3c84b3f5c7bf1ed3e2bf97735cb0"}
Nov 01 08:31:38.612798 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.612661    2586 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T08:31:38.488334763Z","UUID":"3cd9726f-f6a8-4026-8847-571610dbcd77","Handler":null,"Name":"","Endpoint":""}
Nov 01 08:31:38.614834 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.614814    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 08:31:38.614964 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.614843    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 08:31:38.801980 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.801922    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p6sk5" podStartSLOduration=4.50108765 podStartE2EDuration="10.801902538s" totalImagesPullingTime="6.300814888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:35.691537804 +0000 UTC m=+9.662587481" watchObservedRunningTime="2026-11-01 08:31:38.801902538 +0000 UTC m=+12.772952218"
Nov 01 08:31:38.802442 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.802408    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-92bsb" podStartSLOduration=2.956133095 podStartE2EDuration="11.80239913s" totalImagesPullingTime="8.846266035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:38.801357402 +0000 UTC m=+12.772407079" watchObservedRunningTime="2026-11-01 08:31:38.80239913 +0000 UTC m=+12.773449021"
Nov 01 08:31:38.919086 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:38.918818    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sjnbq" podStartSLOduration=2.701181688 podStartE2EDuration="11.918797002s" totalImagesPullingTime="9.217615314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:38.918123919 +0000 UTC m=+12.889173596" watchObservedRunningTime="2026-11-01 08:31:38.918797002 +0000 UTC m=+12.889846679"
Nov 01 08:31:39.490653 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:39.490605    2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:39.491487 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:39.491463    2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:39.593441 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:39.592815    2586 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="00e24beff5c4d4c81ccdb588bccd8012ebbca407505a543d2ea11be87993c286" exitCode=0
Nov 01 08:31:39.593441 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:39.592955    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerDied","Data":"00e24beff5c4d4c81ccdb588bccd8012ebbca407505a543d2ea11be87993c286"}
Nov 01 08:31:40.479147 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:40.479119    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:40.479324 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:40.479155    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:40.479324 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:40.479249    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:40.479410 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:40.479335    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:40.479410 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:40.479395    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:40.479480 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:40.479450    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:40.597174 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:40.597140    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bgtrj" event={"ID":"e0369242-85ce-442a-a759-cd0d52c7fdef","Type":"ContainerStarted","Data":"66671eb5cf4f87521ce184a5d59d6e1c0d019f24afb948a14d78ffd0a0323b6c"}
Nov 01 08:31:40.705219 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:40.705159    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bgtrj" podStartSLOduration=4.304132562 podStartE2EDuration="13.705141753s" totalImagesPullingTime="9.401009191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:40.704571752 +0000 UTC m=+14.675621421" watchObservedRunningTime="2026-11-01 08:31:40.705141753 +0000 UTC m=+14.676191428"
Nov 01 08:31:41.601845 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:41.601810    2586 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="1543999a78a2b260837f16f54394d4c0d5ebcafef5c38b61aa628e58395fc758" exitCode=0
Nov 01 08:31:41.602445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:41.601908    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerDied","Data":"1543999a78a2b260837f16f54394d4c0d5ebcafef5c38b61aa628e58395fc758"}
Nov 01 08:31:41.603678 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:41.603650    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nw2n6" event={"ID":"083acc55-e6ba-4016-abad-67be0cc4af2c","Type":"ContainerStarted","Data":"777226d15e366d5fa0b32e98eb5406d42e572c4132072dec24c057c5ca874668"}
Nov 01 08:31:41.605805 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:41.605780    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg" event={"ID":"55eaf339-45f2-4182-bcab-a01f489180ee","Type":"ContainerStarted","Data":"a0c470ad1589f43222eaf136c7b2532399fe20b21cd5ef039e7c90e47e47fddb"}
Nov 01 08:31:41.877501 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:41.877383    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nw2n6" podStartSLOduration=3.148791125 podStartE2EDuration="14.877364731s" totalImagesPullingTime="11.728573606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:41.877165075 +0000 UTC m=+15.848214747" watchObservedRunningTime="2026-11-01 08:31:41.877364731 +0000 UTC m=+15.848414402"
Nov 01 08:31:42.017659 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:42.017607    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg" podStartSLOduration=2.455694479 podStartE2EDuration="14.0175944s" totalImagesPullingTime="11.561899921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:42.017230915 +0000 UTC m=+15.988280591" watchObservedRunningTime="2026-11-01 08:31:42.0175944 +0000 UTC m=+15.988644066"
Nov 01 08:31:42.479597 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:42.479561    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:42.479773 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:42.479672    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:42.479773 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:42.479572    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:42.479773 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:42.479712    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:42.479879 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:42.479804    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:42.479919 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:42.479879    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:43.612036 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:43.612000    2586 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="6d094b02384d8620431e51331d08dc18dcae5a5b267e09689c9d36e0b1ae03d0" exitCode=0
Nov 01 08:31:43.612589 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:43.612073    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerDied","Data":"6d094b02384d8620431e51331d08dc18dcae5a5b267e09689c9d36e0b1ae03d0"}
Nov 01 08:31:44.372000 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:44.371958    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:44.372206 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:44.372025    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:44.372206 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:44.372065    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:44.372206 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372142    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:44.372206 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372181    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:44.372206 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372204    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:44.372410 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372215    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret podName:3370a9e2-dba0-475a-a84e-3d0d1cfab280 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.372200684 +0000 UTC m=+34.343250352 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret") pod "global-pull-secret-syncer-5cgcj" (UID: "3370a9e2-dba0-475a-a84e-3d0d1cfab280") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:44.372410 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372226    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:44.372410 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372232    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.372224994 +0000 UTC m=+34.343274643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:44.372410 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372240    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:44.372410 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.372299    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.372283917 +0000 UTC m=+34.343333584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:44.479239 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:44.479206    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:44.479391 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:44.479220    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:44.479391 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.479349    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:44.479391 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:44.479363    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:44.479498 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.479424    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:44.479498 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:44.479470    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:45.619727 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:45.619649    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"4adb157789abf8f0befff06f4cfb73ca99785ef67d7eaa17882cdfc36faeeffc"}
Nov 01 08:31:45.620157 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:45.619756    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"1e9e1ed1c8162b19b58f7193bed7e84f3c73635ac0c3c9208284922ec4891da2"}
Nov 01 08:31:45.620157 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:45.619790    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"cfcdba9defa46da9f329b712d8e6ad20977c8562a13ed69288542ff679e0078e"}
Nov 01 08:31:45.620157 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:45.619827    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"03bd06b6bacb1d1ed588979b5a628f203e4d278015b638d7ac7c535a006a5091"}
Nov 01 08:31:45.620157 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:45.619855    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"6a867c86608fc3c4f0a0f61f74ed71da7df2077da0ea2c5927bd05fbb1eadea4"}
Nov 01 08:31:45.620157 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:45.619893    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"2ac0d3fd972b270e1b5042faa66554d67938004c51183b49e6c9558e61870a07"}
Nov 01 08:31:46.479508 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:46.479474    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:46.479686 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:46.479571    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:46.479725 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:46.479682    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:46.479831 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:46.479808    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:46.479862 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:46.479840    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:46.479923 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:46.479910    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:46.624076 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:46.624038    2586 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="010b4badbb93fcb72efca79a01583c87432ca49dd94f6bb21d1c6c053c142f13" exitCode=0
Nov 01 08:31:46.624518 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:46.624089    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerDied","Data":"010b4badbb93fcb72efca79a01583c87432ca49dd94f6bb21d1c6c053c142f13"}
Nov 01 08:31:47.629431 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:47.629397    2586 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="a2c2dc4c6f511186d1fe921349bdffa131c45849f5500a0a860109945fe5623d" exitCode=0
Nov 01 08:31:47.630035 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:47.629481    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerDied","Data":"a2c2dc4c6f511186d1fe921349bdffa131c45849f5500a0a860109945fe5623d"}
Nov 01 08:31:47.632848 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:47.632826    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"002f6094aa399b7743def14d084d4dcc220fc58c0f06fb6663c90426ba40aad6"}
Nov 01 08:31:48.482443 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:48.482410    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:48.482649 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:48.482500    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:48.482649 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:48.482520    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:48.482762 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:48.482639    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:48.482762 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:48.482745    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:48.482848 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:48.482816    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:48.638013 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:48.637983    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cngb" event={"ID":"65196622-05c5-4993-be70-8921dfd83293","Type":"ContainerStarted","Data":"66d465950eec1c268ee91cc0568c75b436b0eb440da6464f583344284bbe873d"}
Nov 01 08:31:48.754407 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:48.754307    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8cngb" podStartSLOduration=7.447802793 podStartE2EDuration="21.754290016s" totalImagesPullingTime="14.306487223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:48.754075522 +0000 UTC m=+22.725125197" watchObservedRunningTime="2026-11-01 08:31:48.754290016 +0000 UTC m=+22.725339688"
Nov 01 08:31:50.479267 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.479234    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:50.479267 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.479250    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:50.479995 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.479250    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:50.479995 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:50.479340    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:50.479995 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:50.479425    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:50.479995 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:50.479519    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:50.644620 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.644583    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" event={"ID":"a7c7f909-3dd3-473a-aacd-356bce893dcb","Type":"ContainerStarted","Data":"ca40612234829f660df7278181e863bd90ad963e77bd0c11af6a78e1e5d3330f"}
Nov 01 08:31:50.644948 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.644921    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:50.660189 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.660169    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:50.907511 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:50.907450    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp" podStartSLOduration=7.654275583 podStartE2EDuration="23.907437816s" totalImagesPullingTime="16.253162233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:50.907111425 +0000 UTC m=+24.878161096" watchObservedRunningTime="2026-11-01 08:31:50.907437816 +0000 UTC m=+24.878487488"
Nov 01 08:31:51.646448 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:51.646421    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:51.646819 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:51.646486    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:51.660882 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:51.660861    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:31:52.346308 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:52.346265    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5cgcj"]
Nov 01 08:31:52.346476 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:52.346400    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:52.346553 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:52.346508    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:52.347163 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:52.347133    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4lxtb"]
Nov 01 08:31:52.347272 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:52.347251    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:52.347371 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:52.347351    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:52.348741 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:52.348720    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkhtc"]
Nov 01 08:31:52.348863 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:52.348824    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:52.348937 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:52.348919    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:53.479724 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:53.479694    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:53.480146 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:53.479700    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:53.480146 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:53.479791    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:53.480146 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:53.479873    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:54.481884 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:54.481858    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:54.482266 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:54.481945    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:55.479480 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:55.479447    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:55.479678 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:55.479593    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:55.479678 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:55.479641    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:55.479797 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:55.479727    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:56.480683 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:56.480652    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:56.481047 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:56.480773    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:57.479759 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:57.479708    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:57.479759 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:57.479738    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:57.480067 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:57.479835    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhtc" podUID="c3286913-2025-4768-8e0f-65467b292ccc"
Nov 01 08:31:57.480067 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:57.479971    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgcj" podUID="3370a9e2-dba0-475a-a84e-3d0d1cfab280"
Nov 01 08:31:58.479176 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.479141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:31:58.479554 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:58.479263    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lxtb" podUID="6483c164-427b-49ee-8a3c-bf576168ed77"
Nov 01 08:31:58.556886 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.556844    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:58.557054 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.556990    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 08:31:58.557485 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.557465    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:31:58.782011 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.781984    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeReady"
Nov 01 08:31:58.782161 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.782114    2586 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 08:31:58.892017 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.891993    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q9w7q"]
Nov 01 08:31:58.934345 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.934314    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:31:58.966564 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.966518    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2bz2q"]
Nov 01 08:31:58.966701 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.966637    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:58.980565 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.980542    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q9w7q"]
Nov 01 08:31:58.980668 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.980582    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bz2q"]
Nov 01 08:31:58.980668 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:58.980618    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.015998 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.015970    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 08:31:59.016141 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.016111    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 08:31:59.016268 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.016254    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h4rfg\""
Nov 01 08:31:59.031089 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.031065    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 08:31:59.075451 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.075382    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25832b5-0080-46b6-a543-f60a7d4da56e-cert\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.075451 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.075415    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343be02-b373-4afb-8a8d-4e2780066e18-tmp-dir\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.075451 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.075433    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.075451 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.075450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt27c\" (UniqueName: \"kubernetes.io/projected/d25832b5-0080-46b6-a543-f60a7d4da56e-kube-api-access-pt27c\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.076003 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.075973    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1343be02-b373-4afb-8a8d-4e2780066e18-config-volume\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.076131 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.076030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w555j\" (UniqueName: \"kubernetes.io/projected/1343be02-b373-4afb-8a8d-4e2780066e18-kube-api-access-w555j\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.116846 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.116807    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 08:31:59.116846 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.116808    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 08:31:59.116846 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.116824    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mnx4s\""
Nov 01 08:31:59.128181 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.128143    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:31:59.177129 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1343be02-b373-4afb-8a8d-4e2780066e18-config-volume\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.177129 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w555j\" (UniqueName: \"kubernetes.io/projected/1343be02-b373-4afb-8a8d-4e2780066e18-kube-api-access-w555j\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.177326 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177162    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25832b5-0080-46b6-a543-f60a7d4da56e-cert\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.177326 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177180    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343be02-b373-4afb-8a8d-4e2780066e18-tmp-dir\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.177326 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177196    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.177326 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177216    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt27c\" (UniqueName: \"kubernetes.io/projected/d25832b5-0080-46b6-a543-f60a7d4da56e-kube-api-access-pt27c\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.177507 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:59.177429    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:31:59.177507 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:59.177504    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.677484536 +0000 UTC m=+33.648534201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:31:59.177843 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.177819    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343be02-b373-4afb-8a8d-4e2780066e18-tmp-dir\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.178070 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.178045    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1343be02-b373-4afb-8a8d-4e2780066e18-config-volume\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.181006 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.180988    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25832b5-0080-46b6-a543-f60a7d4da56e-cert\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.235301 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.235261    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt27c\" (UniqueName: \"kubernetes.io/projected/d25832b5-0080-46b6-a543-f60a7d4da56e-kube-api-access-pt27c\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.237771 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.237745    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w555j\" (UniqueName: \"kubernetes.io/projected/1343be02-b373-4afb-8a8d-4e2780066e18-kube-api-access-w555j\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.314944 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.314912    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:31:59.322919 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.322883    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:31:59.478993 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.478965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:31:59.479144 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.479003    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:31:59.528714 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.528679    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 08:31:59.529039 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.528916    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\""
Nov 01 08:31:59.529039 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.528969    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\""
Nov 01 08:31:59.529039 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.529019    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 08:31:59.680396 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.680325    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:31:59.680544 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:59.680452    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:31:59.680544 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:31:59.680509    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.680494658 +0000 UTC m=+34.651544327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:31:59.760622 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:31:59.760591    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bz2q"]
Nov 01 08:31:59.764329 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:31:59.764302    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25832b5_0080_46b6_a543_f60a7d4da56e.slice/crio-694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543 WatchSource:0}: Error finding container 694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543: Status 404 returned error can't find the container with id 694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543
Nov 01 08:32:00.383980 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.383940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:00.384143 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.383992    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:00.384143 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.384050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:00.384143 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.384090    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:00.384262 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.384155    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:32:32.38413763 +0000 UTC m=+66.355187279 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : secret "metrics-daemon-secret" not found
Nov 01 08:32:00.384262 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.384159    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:32:00.384262 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.384176    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:32:00.384262 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.384188    2586 projected.go:196] Error preparing data for projected volume kube-api-access-mgjlk for pod openshift-network-diagnostics/network-check-target-4lxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:32:00.384262 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.384247    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk podName:6483c164-427b-49ee-8a3c-bf576168ed77 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:32.384223389 +0000 UTC m=+66.355273061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mgjlk" (UniqueName: "kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk") pod "network-check-target-4lxtb" (UID: "6483c164-427b-49ee-8a3c-bf576168ed77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:32:00.386190 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.386170    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:00.401119 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.401097    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:00.479608 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.479581    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:00.529293 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.529268    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 08:32:00.529628 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.529331    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 08:32:00.536388 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.536373    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\""
Nov 01 08:32:00.668480 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.668409    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bz2q" event={"ID":"d25832b5-0080-46b6-a543-f60a7d4da56e","Type":"ContainerStarted","Data":"694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543"}
Nov 01 08:32:00.686747 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.686718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:00.686886 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.686870    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:00.686947 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:00.686936    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.686918588 +0000 UTC m=+36.657968256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:00.813233 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:00.813193    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5cgcj"]
Nov 01 08:32:00.816176 ip-10-0-134-93 kubenswrapper[2586]: W1101 08:32:00.816129    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3370a9e2_dba0_475a_a84e_3d0d1cfab280.slice/crio-91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2 WatchSource:0}: Error finding container 91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2: Status 404 returned error can't find the container with id 91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2
Nov 01 08:32:01.672696 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:01.672660    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5cgcj" event={"ID":"3370a9e2-dba0-475a-a84e-3d0d1cfab280","Type":"ContainerStarted","Data":"91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2"}
Nov 01 08:32:02.702592 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:02.702558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:02.703085 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:02.702725    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:02.703085 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:02.702787    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.702771834 +0000 UTC m=+40.673821497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:02.722560 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:02.722515    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:04.682856 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:04.682277    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bz2q" event={"ID":"d25832b5-0080-46b6-a543-f60a7d4da56e","Type":"ContainerStarted","Data":"9cea1d4db18414a71a9db6cf2408bddc35ce13724bd445e2fe29fb512ec94718"}
Nov 01 08:32:04.792445 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:04.792390    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2bz2q" podStartSLOduration=2.804225208 podStartE2EDuration="6.792370318s" totalImagesPullingTime="3.98814511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:04.7915213 +0000 UTC m=+38.762570969" watchObservedRunningTime="2026-11-01 08:32:04.792370318 +0000 UTC m=+38.763419997"
Nov 01 08:32:06.362115 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.362079    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q4bc2"]
Nov 01 08:32:06.381214 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.381183    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.478470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.478383    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 08:32:06.478470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.478420    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 08:32:06.478470 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.478438    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 08:32:06.478648 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.478477    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 08:32:06.478648 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.478612    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmhw9\""
Nov 01 08:32:06.482219 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.482151    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4bc2"]
Nov 01 08:32:06.521116 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.521090    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:06.531405 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.531386    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.531491 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.531429    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007b6d34-3e90-4550-a5e8-508825a98796-data-volume\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.531491 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.531458    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwn58\" (UniqueName: \"kubernetes.io/projected/007b6d34-3e90-4550-a5e8-508825a98796-kube-api-access-jwn58\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.531580 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.531524    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007b6d34-3e90-4550-a5e8-508825a98796-crio-socket\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.531613 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.531579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.616186 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.616107    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5m778-tsfb9"]
Nov 01 08:32:06.631989 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.631951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.632125 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.632029    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007b6d34-3e90-4550-a5e8-508825a98796-data-volume\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.632125 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.632060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwn58\" (UniqueName: \"kubernetes.io/projected/007b6d34-3e90-4550-a5e8-508825a98796-kube-api-access-jwn58\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.632125 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:06.632073    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:06.632125 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.632090    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007b6d34-3e90-4550-a5e8-508825a98796-crio-socket\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.632125 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.632115    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.632339 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:06.632141    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:07.132121958 +0000 UTC m=+41.103171628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:06.632339 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:06.632200    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:06.632339 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:06.632256    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:07.132240285 +0000 UTC m=+41.103289949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:06.632339 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.632285    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007b6d34-3e90-4550-a5e8-508825a98796-crio-socket\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.632488 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.632419    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007b6d34-3e90-4550-a5e8-508825a98796-data-volume\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:06.635058 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.635041    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:06.674047 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.674014    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5m778-tsfb9"]
Nov 01 08:32:06.685574 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.685552    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q5m6c\""
Nov 01 08:32:06.685574 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.685565    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 08:32:06.688522 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.688495    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5cgcj" event={"ID":"3370a9e2-dba0-475a-a84e-3d0d1cfab280","Type":"ContainerStarted","Data":"51e3e0dd846b916f5e79454a55ef1a9b1d24eb29b99e4fa879886ad08320543d"}
Nov 01 08:32:06.726716 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.726686    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:06.732526 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.732509    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:06.732639 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.732548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:06.732697 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:06.732671    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:06.732697 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.732675    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:06.732797 ip-10-0-134-93 kubenswrapper[2586]: E1101 08:32:06.732782    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.732750773 +0000 UTC m=+48.703800423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:06.732860 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.732846    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:06.732892 ip-10-0-134-93 kubenswrapper[2586]: I1101 08:32:06.732868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcht\" (UniqueName: \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:06.747975 ip-10-0-134-93 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 08:32:06.764977 ip-10-0-134-93 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 08:32:06.765227 ip-10-0-134-93 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 08:32:06.765295 ip-10-0-134-93 systemd[1]: kubelet.service: Consumed 2.256s CPU time.
Nov 01 08:32:06.777149 ip-10-0-134-93 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 08:32:06.849009 ip-10-0-134-93 kubenswrapper[4314]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:32:06.849009 ip-10-0-134-93 kubenswrapper[4314]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 08:32:06.849009 ip-10-0-134-93 kubenswrapper[4314]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:32:06.849009 ip-10-0-134-93 kubenswrapper[4314]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:32:06.851004 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.850935    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.851004 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851004    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851008    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851012    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851015    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851017    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851021    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851024    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851032    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851035    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851038    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851041    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851043    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851046    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851049    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851052    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851055    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851058    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851060    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851063    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.851071 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851066    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851069    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851071    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851074    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851077    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851080    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851083    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851086    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851088    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851091    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851094    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851096    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851099    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851102    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851104    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851107    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851109    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851112    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851115    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.851562 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851118    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851120    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851123    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851127    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851131    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851134    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851137    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851140    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851142    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851145    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851147    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851150    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851153    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851156    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851159    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851161    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851164    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851167    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851169    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.852026 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851172    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851175    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851177    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851180    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851182    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851185    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851188    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851190    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851193    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851196    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851198    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851201    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851204    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851206    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851215    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851218    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851220    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851223    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851226    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851228    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.852499 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851231    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851233    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851236    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851239    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851242    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851245    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851248    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851250    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851253    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851256    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851258    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851261    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851264    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851267    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851271    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851275    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851278    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851281    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851284    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851286    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.852998 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851289    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851291    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851294    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851296    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851299    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851301    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851304    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.851314    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852407    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852417    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852420    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852423    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852426    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852428    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852431    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852434    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852438    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852440    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852443    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.853471 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852446    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852449    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852451    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852454    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852457    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852460    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852463    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852465    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852468    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852471    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852474    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852477    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852480    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852482    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852485    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852488    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852490    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852493    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852495    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852498    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.853937 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852501    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852504    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852507    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852510    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852512    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852515    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852519    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852522    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852524    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852527    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852544    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852547    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852550    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852553    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852555    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852558    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852561    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852563    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852566    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.854422 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852569    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852571    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852574    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852577    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852579    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852582    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852585    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852587    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852590    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852592    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852595    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852597    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852600    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852603    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852606    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852612    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852615    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852618    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852621    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852624    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.854948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852627    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852629    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852632    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852634    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852637    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852640    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852643    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852646    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852648    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852651    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852653    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852656    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852658    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852661    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852664    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852666    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852669    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852671    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852674    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852677    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.855427 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852679    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852682    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852684    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852687    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852690    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852693    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852695    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852699    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852709    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852713    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852715    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852718    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852721    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852723    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852726    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.852731    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853209    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853215    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853219    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.855921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853222    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853225    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853227    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853230    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853232    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853235    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853238    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853241    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853243    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853246    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853248    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853251    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853254    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853257    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853260    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853263    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853280    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853284    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853289    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853292    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.856386 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853294    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853297    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853300    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853303    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853306    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853308    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853311    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853315    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853318    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853321    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853325    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853327    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853330    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853333    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853336    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853339    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853341    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853345    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.856921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853349    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853352    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853354    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853357    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853360    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853362    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853365    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853368    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853370    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853373    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853376    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853378    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853381    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853384    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853386    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853389    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853392    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853395    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853397    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853400    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.857357 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853402    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853405    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853407    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853410    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853412    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853415    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853417    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853420    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853422    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853425    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853427    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853430    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853432    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853435    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853437    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853440    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853443    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853446    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853448    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853451    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.857851 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853454    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853456    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853459    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853462    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853467    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853471    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853475    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853479    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853481    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853485    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853488    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853490    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853493    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853496    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853499    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853501    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853505    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853508    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853511    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.858333 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853514    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853516    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853519    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853522    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853524    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.853527    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853622    4314 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853629    4314 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853636    4314 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853641    4314 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853645    4314 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853648    4314 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853652    4314 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853656    4314 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853662    4314 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853665    4314 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853669    4314 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853672    4314 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853675    4314 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853678    4314 flags.go:64] FLAG: --cgroup-root=""
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853681    4314 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853685    4314 flags.go:64] FLAG: --client-ca-file=""
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853687    4314 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 08:32:06.858848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853691    4314 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853696    4314 flags.go:64] FLAG: --cluster-domain=""
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853698    4314 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853702    4314 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853705    4314 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853709    4314 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853713    4314 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853716    4314 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853719    4314 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853722    4314 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853725    4314 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853728    4314 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853731    4314 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853734    4314 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853738    4314 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853742    4314 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853745    4314 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853748    4314 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853751    4314 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853754    4314 flags.go:64] FLAG: --enable-server="true"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853757    4314 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853762    4314 flags.go:64] FLAG: --event-burst="100"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853765    4314 flags.go:64] FLAG: --event-qps="50"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853770    4314 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 08:32:06.859769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853773    4314 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853776    4314 flags.go:64] FLAG: --eviction-hard=""
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853780    4314 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853783    4314 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853786    4314 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853789    4314 flags.go:64] FLAG: --eviction-soft=""
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853792    4314 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853795    4314 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853798    4314 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853802    4314 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853805    4314 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853808    4314 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853811    4314 flags.go:64] FLAG: --feature-gates=""
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853815    4314 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853818    4314 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853821    4314 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853824    4314 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853827    4314 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853830    4314 flags.go:64] FLAG: --help="false"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853833    4314 flags.go:64] FLAG: --hostname-override="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853837    4314 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853840    4314 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853843    4314 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 08:32:06.860516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853846    4314 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853850    4314 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853853    4314 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853856    4314 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853859    4314 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853862    4314 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853865    4314 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853868    4314 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853872    4314 flags.go:64] FLAG: --kube-reserved=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853875    4314 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853878    4314 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853883    4314 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853886    4314 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853889    4314 flags.go:64] FLAG: --lock-file=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853892    4314 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853895    4314 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853898    4314 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853903    4314 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853906    4314 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853910    4314 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853913    4314 flags.go:64] FLAG: --logging-format="text"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853916    4314 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853920    4314 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853923    4314 flags.go:64] FLAG: --manifest-url=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853926    4314 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 08:32:06.861090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853930    4314 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853933    4314 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853941    4314 flags.go:64] FLAG: --max-pods="110"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853946    4314 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853950    4314 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853955    4314 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853959    4314 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853962    4314 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853965    4314 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853969    4314 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853976    4314 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853981    4314 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853986    4314 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853989    4314 flags.go:64] FLAG: --pod-cidr=""
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853992    4314 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853996    4314 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.853999    4314 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854002    4314 flags.go:64] FLAG: --port="10250"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854007    4314 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854010    4314 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0acdb4ce1cd176a56"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854013    4314 flags.go:64] FLAG: --qos-reserved=""
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854016    4314 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854019    4314 flags.go:64] FLAG: --register-node="true"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854022    4314 flags.go:64] FLAG: --register-with-taints=""
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854026    4314 flags.go:64] FLAG: --registry-burst="10"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854029    4314 flags.go:64] FLAG: --registry-qps="5"
Nov 01 08:32:06.861714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854032    4314 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854035    4314 flags.go:64] FLAG: --reserved-memory=""
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854040    4314 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854043    4314 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854046    4314 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854049    4314 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854052    4314 flags.go:64] FLAG: --runonce="false"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854054    4314 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854058    4314 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854061    4314 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854063    4314 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854066    4314 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854069    4314 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854072    4314 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854076    4314 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854079    4314 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854082    4314 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854085    4314 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854088    4314 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854091    4314 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854094    4314 flags.go:64] FLAG: --system-cgroups=""
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854097    4314 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854102    4314 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854105    4314 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854109    4314 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 08:32:06.862338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854113    4314 flags.go:64] FLAG: --tls-min-version=""
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854116    4314 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854118    4314 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854121    4314 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854125    4314 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854128    4314 flags.go:64] FLAG: --v="2"
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854132    4314 flags.go:64] FLAG: --version="false"
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854135    4314 flags.go:64] FLAG: --vmodule=""
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854140    4314 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854144    4314 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854282    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854286    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854290    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854293    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854296    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854299    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854302    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854306    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854308    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854311    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854314    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.862940 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854317    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854320    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854322    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854325    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854328    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854330    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854333    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854336    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854339    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854341    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854346    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854349    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854351    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854355    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854359    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854361    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854364    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854367    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854369    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.863450 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854372    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854375    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854379    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854382    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854384    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854387    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854390    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854394    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854398    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854401    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854404    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854406    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854409    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854411    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854414    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854417    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854420    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854422    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854425    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854428    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.863934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854431    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854434    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854436    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854442    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854445    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854447    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854450    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854454    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854457    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854460    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854462    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854465    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854467    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854470    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854473    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854482    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854485    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854488    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854490    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854493    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.864428 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854496    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854498    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854501    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854504    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854506    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854509    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854512    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854514    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854517    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854519    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854522    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854525    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854527    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854546    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854550    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854555    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854558    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854560    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854563    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854567    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.865093 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854569    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854572    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854575    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854577    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854580    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854583    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854586    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854588    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854592    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854595    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854597    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854600    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854602    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854605    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854608    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854610    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.865759 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.854637    4314 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854681    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854686    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854690    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854692    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854695    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854698    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854701    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854704    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854706    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854709    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854712    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854716    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854718    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854722    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854729    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854732    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854735    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854738    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854740    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.866220 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854743    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854745    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854748    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854751    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854754    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854757    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854760    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854763    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854766    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854768    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854771    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854774    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854776    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854779    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854781    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854784    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854786    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854789    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854791    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854794    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.866835 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854797    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854799    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854802    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854804    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854808    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854811    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854814    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854818    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854821    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854823    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854826    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854829    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854832    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854834    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854837    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854840    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854843    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854846    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854851    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.867365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854854    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854857    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854859    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854862    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854864    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854867    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854870    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854872    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854875    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854877    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854880    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854883    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854885    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854888    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854890    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854893    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854896    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854902    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854906    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.868028 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854909    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854913    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854915    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854918    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854921    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854925    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854927    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854930    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854933    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854936    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854939    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854943    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854945    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854949    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854952    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854955    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854958    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854961    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854963    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.868655 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854966    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854969    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854972    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854975    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854979    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854983    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854986    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854989    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854991    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.854994    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.860396    4314 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.860416    4314 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.860434    4314 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.860441    4314 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860497    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860502    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860505    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860508    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860511    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.869231 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860514    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860516    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860519    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860522    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860526    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860546    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860550    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860553    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860556    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860559    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860562    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860565    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860568    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860570    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860573    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860576    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860579    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860583    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860587    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860591    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.869714 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860596    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860600    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860604    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860606    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860610    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860612    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860615    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860618    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860620    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860623    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860626    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860628    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860632    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860636    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860640    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860644    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860648    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860651    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860654    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.870195 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860657    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860659    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860664    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860666    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860669    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860672    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860674    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860677    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860679    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860682    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860685    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860687    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860690    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860693    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860696    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860699    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860702    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860704    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860707    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860710    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.870764 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860712    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860715    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860718    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860720    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860723    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860725    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860728    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860731    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860733    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860736    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860738    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860741    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860744    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860746    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860749    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860752    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860755    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860757    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860760    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860763    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.871364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860765    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860768    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860770    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860773    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860776    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860779    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860782    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860785    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860787    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860790    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860792    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860795    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860798    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860800    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860803    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860805    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860808    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860810    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860813    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.872021 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860815    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860818    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860821    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.860851    4314 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860985    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860990    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860993    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860997    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.860999    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861003    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861006    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861008    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861011    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861014    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861017    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861019    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.872525 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861022    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861024    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861027    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861030    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861032    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861035    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861038    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861040    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861043    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861045    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861048    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861051    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861053    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861056    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861059    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861061    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861064    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861066    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861069    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861072    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.872929 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861075    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861078    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861080    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861083    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861085    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861088    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861091    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861094    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861097    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861100    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861103    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861105    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861108    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861111    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861114    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861116    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861119    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861121    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861124    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.873404 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861126    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861129    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861132    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861134    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861137    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861139    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861142    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861145    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861147    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861150    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861152    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861155    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861158    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861160    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861163    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861165    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861168    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861171    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861174    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861176    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.873871 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861179    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861183    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861187    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861190    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861193    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861196    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861198    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861201    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861204    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861207    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861209    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861212    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861214    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861217    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861220    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861223    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861225    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861228    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861231    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.874374 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861233    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861236    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861238    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861241    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861246    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861249    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861252    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861255    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861258    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861260    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861264    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861267    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861270    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861272    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861276    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861278    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.874863 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.861302    4314 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861334    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861337    4314 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861341    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861344    4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861346    4314 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861349    4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861352    4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861355    4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861357    4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861360    4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861362    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861365    4314 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861368    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861371    4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861373    4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861376    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861378    4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861381    4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861384    4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:06.875248 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861388    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861391    4314 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861393    4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861396    4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861399    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861401    4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861404    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861407    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861409    4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861412    4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861415    4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861418    4314 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861421    4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861423    4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861426    4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861428    4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861431    4314 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861433    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861436    4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:06.875746 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861439    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861442    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861444    4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861447    4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861449    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861452    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861455    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861457    4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861460    4314 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861465    4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861468    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861471    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861474    4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861478    4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861481    4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861483    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861486    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861489    4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861491    4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:06.876266 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861494    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861496    4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861499    4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861502    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861504    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861507    4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861510    4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861512    4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861515    4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861518    4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861520    4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861523    4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861525    4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861541    4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861546    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861549    4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861552    4314 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861554    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861558    4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:06.876759 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861562    4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861565    4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861567    4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861570    4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861573    4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861575    4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861578    4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861583    4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861586    4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861588    4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861591    4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861594    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861596    4314 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861599    4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861601    4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861604    4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861607    4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861609    4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861612    4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861616    4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:06.877456 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861619    4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861622    4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861625    4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861628    4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861631    4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861633    4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861636    4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861639    4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861642    4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:06.861645    4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.861700    4314 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.862652    4314 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.862718    4314 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.862980    4314 server.go:1014] "Starting client certificate rotation"
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.863120    4314 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.863178    4314 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 08:26:27 +0000 UTC" deadline="2028-07-26 17:22:46.014141846 +0000 UTC"
Nov 01 08:32:06.878229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.863205    4314 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15200h50m39.150938718s"
Nov 01 08:32:06.878859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.863487    4314 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:32:06.878859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.864501    4314 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:32:06.878859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.865841    4314 log.go:25] "Validated CRI v1 runtime API"
Nov 01 08:32:06.878859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.867239    4314 log.go:25] "Validated CRI v1 image API"
Nov 01 08:32:06.878859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.867964    4314 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 08:32:06.878859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.869889    4314 fs.go:135] Filesystem UUIDs: map[05f5aa4e-a328-4f35-9380-895bf6cc8375:/dev/nvme0n1p3 62222a2b-8911-4ad7-a74d-f4a600405243:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 08:32:06.879123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.869912    4314 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~projected/kube-api-access-bwqvp:{mountpoint:/var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~projected/kube-api-access-bwqvp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1343be02-b373-4afb-8a8d-4e2780066e18/volumes/kubernetes.io~projected/kube-api-access-w555j:{mountpoint:/var/lib/kubelet/pods/1343be02-b373-4afb-8a8d-4e2780066e18/volumes/kubernetes.io~projected/kube-api-access-w555j major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3370a9e2-dba0-475a-a84e-3d0d1cfab280/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3370a9e2-dba0-475a-a84e-3d0d1cfab280/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c33083a-5961-4854-a5af-8b1de96e6eec/volumes/kubernetes.io~projected/kube-api-access-8b5fr:{mountpoint:/var/lib/kubelet/pods/3c33083a-5961-4854-a5af-8b1de96e6eec/volumes/kubernetes.io~projected/kube-api-access-8b5fr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47ad972a-7efb-4755-bdae-e2087f130fb4/volumes/kubernetes.io~projected/kube-api-access-62767:{mountpoint:/var/lib/kubelet/pods/47ad972a-7efb-4755-bdae-e2087f130fb4/volumes/kubernetes.io~projected/kube-api-access-62767 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55eaf339-45f2-4182-bcab-a01f489180ee/volumes/kubernetes.io~projected/kube-api-access-hs477:{mountpoint:/var/lib/kubelet/pods/55eaf339-45f2-4182-bcab-a01f489180ee/volumes/kubernetes.io~projected/kube-api-access-hs477 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65196622-05c5-4993-be70-8921dfd83293/volumes/kubernetes.io~projected/kube-api-access-n2bsq:{mountpoint:/var/lib/kubelet/pods/65196622-05c5-4993-be70-8921dfd83293/volumes/kubernetes.io~projected/kube-api-access-n2bsq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volumes/kubernetes.io~projected/kube-api-access-ztfq5:{mountpoint:/var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volumes/kubernetes.io~projected/kube-api-access-ztfq5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bade5291-3729-42d3-92da-7fe9e224a4ad/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bade5291-3729-42d3-92da-7fe9e224a4ad/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3286913-2025-4768-8e0f-65467b292ccc/volumes/kubernetes.io~projected/kube-api-access-s6czd:{mountpoint:/var/lib/kubelet/pods/c3286913-2025-4768-8e0f-65467b292ccc/volumes/kubernetes.io~projected/kube-api-access-s6czd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9b827d2-1aac-4471-9665-21c2575f6d0c/volumes/kubernetes.io~projected/kube-api-access-j6cqz:{mountpoint:/var/lib/kubelet/pods/c9b827d2-1aac-4471-9665-21c2575f6d0c/volumes/kubernetes.io~projected/kube-api-access-j6cqz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25832b5-0080-46b6-a543-f60a7d4da56e/volumes/kubernetes.io~projected/kube-api-access-pt27c:{mountpoint:/var/lib/kubelet/pods/d25832b5-0080-46b6-a543-f60a7d4da56e/volumes/kubernetes.io~projected/kube-api-access-pt27c major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25832b5-0080-46b6-a543-f60a7d4da56e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d25832b5-0080-46b6-a543-f60a7d4da56e/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0369242-85ce-442a-a759-cd0d52c7fdef/volumes/kubernetes.io~projected/kube-api-access-jmjbx:{mountpoint:/var/lib/kubelet/pods/e0369242-85ce-442a-a759-cd0d52c7fdef/volumes/kubernetes.io~projected/kube-api-access-jmjbx major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7a2a021d43f28fe99b300feaa4ad0f2e08b50dcc2254d72b8a2074141ea152eb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ebba11ccceb5ab6cde7cfb34d2c357a545591395ffbb9a31e5648165f77c78a1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5c1e63d7906a12e618494d925f092c9f7c945fce17ae5aa2404aa60f654e0548/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/67e2b1ff4df738bf8f2038af62098be4f7dc119236b92e1ca3a61fb770eed495/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/37d36a51ae3886b6e08662d3d32618133bfe6c626cc8fec5b349dc1b3f644267/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4fa407867cb287be4f7d3d94a1fe931ebad3b789b9eaabb38a1b4aa08178d3c0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3428a985e60df49af0dfbf954aefe9a77f9b356a4c777d11b649adab638f5471/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/21532fb47cb8ed59f6a763218fbde7b12eb8eaaf66856bb89edeb77cea974e6e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5850a8bb7125661739d1ad0d642d516b2affce5cec575893c2d771a25ea3aa83/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1ec118b42db1fc02e2d984c616031466ce7bccf18c7c84c51019a1507cedcb69/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/28e943ddc437a942b5c28d567482bacd2b137ea82f6644c8823cc70ca691232f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/8661d70d8fc56bf8b722b0afea6c9499d77931610bb0b05bd35d48f70fc2d128/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/611148b00cb46d909c93d2b646ad71f352744319b181b0c6ca89b4c03bc581ca/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/bf51c67dd9d915fccbfa2bb5886dde86cd445502415ed6bbb1601f50deca6ec0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cf9e941f0c7e3e826eaa634bd18268c309ada8ee2692f7f64afc89c40c36c436/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/aca3540e2f499dacf2ecdefa2f2a68e4508d3b95a700d412d42d8858013b6b30/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ee5e8cbf333b7740828a3753fdfa76624137e40a1c954029bd05d8ca2307663a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d23bc4075e4f47e1fa95282b51a1f099b923a9ad3f8a465df9ee2f897711c4bf/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/039e2ce48742773283b98c1d9dee89fceaccc77e836c5bf68e85099e2a941a9a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d7d3dcb86c48169d97fce9d9207b1b7e6ba35ed2ebbd7ca13a0c8a6b88c4395c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/197b19c9979a5030e2690a20d393fbdcd7932407ca85f65fca6c2f16966f9760/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/32fd6cff1356f659f1b5cf184baa6e1e21004271ae0bc6b4f440d86cbf3da0f0/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/40044bd542e911efd0a9fda73d5de417674975955289adf6a1a05f6cafc54a10/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c2d96edc29618bf2974557bbadc7e9567b4671b77724d9ec63c6ed90ff890ef4/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f420b2acd96a1628003288927ea2c59805ae1034a96ec4b1e2904aa878f4aa7a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/59e2377b5d0203ca29fd86acf2fe8f75ef714c5f31c19a32725bd718913dcf2c/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/22c479024d311a29c8f1306c1144e5091c7a8b141d46bb5407f7a2761cebb7c5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a50586fc3fce7ec95c46e25b5f49cbc48a7f950f2c906a7551b0c3c4614a214f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/4ad1bfc81006a4c289cd37a26cb804c4fa88ef40624f04493ce51da17c0978f7/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a1e6dc8b5e5785d9fcfc7a670cc3a6db70cacf1c1ad68f7092b50b813746408f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/11e1e64606b04380e91a0750b46016aa3523a428d14d3c588e3070b72bf47ac3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/408596b77a5f85d48b5cc2300c2e8cb019c509434d7e6da802ead163354a248e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/64ac37c82fc259655dbf828f04d97d75a6347bce900b28d2cf6ca4e167c1b16e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/cad74feee4f6e69f6e3fbc087230b030224aef5d0a023a5d329851ffbd96e3fd/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1d93413bc50863e5316e04fdf6dc1d726a050ff61bba6b9966e59da5688c14d3/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 08:32:06.879123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.876453    4314 manager.go:217] Machine: {Timestamp:2026-11-01 08:32:06.875898968 +0000 UTC m=+0.070293515 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100708 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f8d6e108aecf81e6138cf01320294 SystemUUID:ec2f8d6e-108a-ecf8-1e61-38cf01320294 BootID:3096594e-19aa-40f7-a9f7-03f6acf79c9e Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f7b92b4d528cdb3a7f6b17fa75054244c5675a58b58dd58a0ac97feda9b8af9/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/bade5291-3729-42d3-92da-7fe9e224a4ad/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8248768ec2ccec20f04ac2d1b26635fab457cdf50181e020c90fea4e9f4228c0/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d25832b5-0080-46b6-a543-f60a7d4da56e/volumes/kubernetes.io~projected/kube-api-access-pt27c DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47ad972a-7efb-4755-bdae-e2087f130fb4/volumes/kubernetes.io~projected/kube-api-access-62767 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91677c98de496e3c3f0ad0b5e77638ef4b7a5f7adabf7513e371ea8349f3b6a2/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c5fd742ca355a68faac123f7e1f224b0fc728d7c95fcc24523ca499a0fc26c1/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3286913-2025-4768-8e0f-65467b292ccc/volumes/kubernetes.io~projected/kube-api-access-s6czd DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4417d278a9729d0bf53adb3473c27cdb41873f36afaa0f6ae96a11a3e98b7cdb/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/039f83153243c63a9bba2ad3ca76bdc8e05310b833183906e4f19e83461463ad/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5eb916eab82263aea7884d79f5da6437c079a125842a68ed69587785f04dc4a/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0369242-85ce-442a-a759-cd0d52c7fdef/volumes/kubernetes.io~projected/kube-api-access-jmjbx DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3370a9e2-dba0-475a-a84e-3d0d1cfab280/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9b827d2-1aac-4471-9665-21c2575f6d0c/volumes/kubernetes.io~projected/kube-api-access-j6cqz DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/769df45642571c52f693bee595d45d9c47bd54fe56930eea1ddb74ff7757e8bd/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55eaf339-45f2-4182-bcab-a01f489180ee/volumes/kubernetes.io~projected/kube-api-access-hs477 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/694c5602b9988cf18a96f70ab34aa440065683fc09369e5d3a7f72462afe7543/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ec277cba7e5159b3432f35a570fe31440bb7939d2eeceaa33653ed0c0291a28/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1343be02-b373-4afb-8a8d-4e2780066e18/volumes/kubernetes.io~projected/kube-api-access-w555j DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volumes/kubernetes.io~projected/kube-api-access-ztfq5 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d25832b5-0080-46b6-a543-f60a7d4da56e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/083acc55-e6ba-4016-abad-67be0cc4af2c/volumes/kubernetes.io~projected/kube-api-access-bwqvp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3c33083a-5961-4854-a5af-8b1de96e6eec/volumes/kubernetes.io~projected/kube-api-access-8b5fr DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c7f909-3dd3-473a-aacd-356bce893dcb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65196622-05c5-4993-be70-8921dfd83293/volumes/kubernetes.io~projected/kube-api-access-n2bsq DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:694c5602b9988cf MacAddress:ee:32:e7:94:75:0b Speed:10000 Mtu:8901} {Name:91677c98de496e3 MacAddress:36:be:51:7f:02:d7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ce:f8:59:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:cc:36:93:3e:36 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:ce:f8:59:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e2:e4:8d:4c:23:0e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:93:3e:30:2b:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 08:32:06.879123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.876709    4314 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 08:32:06.879123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.876760    4314 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 08:32:06.879123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877098    4314 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 08:32:06.879123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877118    4314 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-93.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877247    4314 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877255    4314 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877265    4314 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877287    4314 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877363    4314 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877556    4314 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877667    4314 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877706    4314 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877715    4314 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877725    4314 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877734    4314 kubelet.go:405] "Adding apiserver pod source"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.877742    4314 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.878991    4314 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 08:32:06.880079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.879053    4314 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:32:06.880519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.880241    4314 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 08:32:06.882851 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.882833    4314 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 08:32:06.882851 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.882848    4314 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:32:06.882995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.882945    4314 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 08:32:06.883139 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883125    4314 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 08:32:06.883184 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883156    4314 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 08:32:06.883256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883247    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 08:32:06.883294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883261    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 08:32:06.883294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883268    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 08:32:06.883294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883281    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 08:32:06.883294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883288    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 08:32:06.883294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883294    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883300    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883306    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883313    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883318    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883327    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883337    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883359    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 08:32:06.883481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.883365    4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 08:32:06.886521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.886508    4314 server.go:1263] "Started kubelet"
Nov 01 08:32:06.886725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.886619    4314 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 08:32:06.887028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.886963    4314 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 08:32:06.887284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.887048    4314 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 08:32:06.887041 ip-10-0-134-93 systemd[1]: Started Kubernetes Kubelet.
Nov 01 08:32:06.887832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.887812    4314 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 08:32:06.888323 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888305    4314 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 08:32:06.888867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888850    4314 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 08:32:06.889006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888868    4314 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 08:32:06.889006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888963    4314 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:29 +0000 UTC" deadline="2028-04-13 14:33:56.214444569 +0000 UTC"
Nov 01 08:32:06.889006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888980    4314 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12702h1m49.325467431s"
Nov 01 08:32:06.889006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888983    4314 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 08:32:06.889006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.888967    4314 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 08:32:06.889006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.889007    4314 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 08:32:06.889615 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:06.889093    4314 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:32:06.890361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890320    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c33083a-5961-4854-a5af-8b1de96e6eec" volumeName="kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-cni-binary-copy" seLinuxMountContext=""
Nov 01 08:32:06.890426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890366    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c33083a-5961-4854-a5af-8b1de96e6eec" volumeName="kubernetes.io/projected/3c33083a-5961-4854-a5af-8b1de96e6eec-kube-api-access-8b5fr" seLinuxMountContext=""
Nov 01 08:32:06.890426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890379    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c7f909-3dd3-473a-aacd-356bce893dcb" volumeName="kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-env-overrides" seLinuxMountContext=""
Nov 01 08:32:06.890426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890390    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3286913-2025-4768-8e0f-65467b292ccc" volumeName="kubernetes.io/projected/c3286913-2025-4768-8e0f-65467b292ccc-kube-api-access-s6czd" seLinuxMountContext=""
Nov 01 08:32:06.890426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890401    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0369242-85ce-442a-a759-cd0d52c7fdef" volumeName="kubernetes.io/projected/e0369242-85ce-442a-a759-cd0d52c7fdef-kube-api-access-jmjbx" seLinuxMountContext=""
Nov 01 08:32:06.890426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890413    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083acc55-e6ba-4016-abad-67be0cc4af2c" volumeName="kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-tmp" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890425    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c7f909-3dd3-473a-aacd-356bce893dcb" volumeName="kubernetes.io/secret/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890439    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c33083a-5961-4854-a5af-8b1de96e6eec" volumeName="kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-daemon-config" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890454    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c7f909-3dd3-473a-aacd-356bce893dcb" volumeName="kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-config" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890465    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9b827d2-1aac-4471-9665-21c2575f6d0c" volumeName="kubernetes.io/empty-dir/c9b827d2-1aac-4471-9665-21c2575f6d0c-tmp-dir" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890476    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083acc55-e6ba-4016-abad-67be0cc4af2c" volumeName="kubernetes.io/projected/083acc55-e6ba-4016-abad-67be0cc4af2c-kube-api-access-bwqvp" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890499    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1343be02-b373-4afb-8a8d-4e2780066e18" volumeName="kubernetes.io/configmap/1343be02-b373-4afb-8a8d-4e2780066e18-config-volume" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890511    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1343be02-b373-4afb-8a8d-4e2780066e18" volumeName="kubernetes.io/empty-dir/1343be02-b373-4afb-8a8d-4e2780066e18-tmp-dir" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890522    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65196622-05c5-4993-be70-8921dfd83293" volumeName="kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890564    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c7f909-3dd3-473a-aacd-356bce893dcb" volumeName="kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890577    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55eaf339-45f2-4182-bcab-a01f489180ee" volumeName="kubernetes.io/projected/55eaf339-45f2-4182-bcab-a01f489180ee-kube-api-access-hs477" seLinuxMountContext=""
Nov 01 08:32:06.890605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890589    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9b827d2-1aac-4471-9665-21c2575f6d0c" volumeName="kubernetes.io/projected/c9b827d2-1aac-4471-9665-21c2575f6d0c-kube-api-access-j6cqz" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890613    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25832b5-0080-46b6-a543-f60a7d4da56e" volumeName="kubernetes.io/projected/d25832b5-0080-46b6-a543-f60a7d4da56e-kube-api-access-pt27c" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890626    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1343be02-b373-4afb-8a8d-4e2780066e18" volumeName="kubernetes.io/projected/1343be02-b373-4afb-8a8d-4e2780066e18-kube-api-access-w555j" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890637    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ad972a-7efb-4755-bdae-e2087f130fb4" volumeName="kubernetes.io/projected/47ad972a-7efb-4755-bdae-e2087f130fb4-kube-api-access-62767" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890649    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65196622-05c5-4993-be70-8921dfd83293" volumeName="kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890661    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade5291-3729-42d3-92da-7fe9e224a4ad" volumeName="kubernetes.io/configmap/bade5291-3729-42d3-92da-7fe9e224a4ad-konnectivity-ca" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890673    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0369242-85ce-442a-a759-cd0d52c7fdef" volumeName="kubernetes.io/configmap/e0369242-85ce-442a-a759-cd0d52c7fdef-iptables-alerter-script" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890686    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="007b6d34-3e90-4550-a5e8-508825a98796" volumeName="kubernetes.io/empty-dir/007b6d34-3e90-4550-a5e8-508825a98796-data-volume" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890697    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65196622-05c5-4993-be70-8921dfd83293" volumeName="kubernetes.io/projected/65196622-05c5-4993-be70-8921dfd83293-kube-api-access-n2bsq" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890707    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c7f909-3dd3-473a-aacd-356bce893dcb" volumeName="kubernetes.io/projected/a7c7f909-3dd3-473a-aacd-356bce893dcb-kube-api-access-ztfq5" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890718    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bade5291-3729-42d3-92da-7fe9e224a4ad" volumeName="kubernetes.io/secret/bade5291-3729-42d3-92da-7fe9e224a4ad-agent-certs" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890730    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25832b5-0080-46b6-a543-f60a7d4da56e" volumeName="kubernetes.io/secret/d25832b5-0080-46b6-a543-f60a7d4da56e-cert" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890749    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083acc55-e6ba-4016-abad-67be0cc4af2c" volumeName="kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-tuned" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890760    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3370a9e2-dba0-475a-a84e-3d0d1cfab280" volumeName="kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890779    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ad972a-7efb-4755-bdae-e2087f130fb4" volumeName="kubernetes.io/configmap/47ad972a-7efb-4755-bdae-e2087f130fb4-serviceca" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890791    4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65196622-05c5-4993-be70-8921dfd83293" volumeName="kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-binary-copy" seLinuxMountContext=""
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890802    4314 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 08:32:06.890973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.890809    4314 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 08:32:06.892711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.892693    4314 factory.go:55] Registering systemd factory
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.892716    4314 factory.go:223] Registration of the systemd container factory successfully
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.892933    4314 factory.go:153] Registering CRI-O factory
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.892947    4314 factory.go:223] Registration of the crio container factory successfully
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.892998    4314 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.893016    4314 factory.go:103] Registering Raw factory
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.893031    4314 manager.go:1199] Started watching for new ooms in manager
Nov 01 08:32:06.893674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.893330    4314 manager.go:319] Starting recovery of all containers
Nov 01 08:32:06.918307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.917874    4314 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 08:32:06.920716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.920691    4314 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 08:32:06.920797 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.920722    4314 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 08:32:06.920797 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.920746    4314 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 08:32:06.920902 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:06.920788    4314 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 08:32:06.931779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.931752    4314 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="a2c2dc4c6f511186d1fe921349bdffa131c45849f5500a0a860109945fe5623d" exitCode=0
Nov 01 08:32:06.931779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.931776    4314 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="010b4badbb93fcb72efca79a01583c87432ca49dd94f6bb21d1c6c053c142f13" exitCode=0
Nov 01 08:32:06.931779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.931784    4314 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="6d094b02384d8620431e51331d08dc18dcae5a5b267e09689c9d36e0b1ae03d0" exitCode=0
Nov 01 08:32:06.931779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.931789    4314 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="1543999a78a2b260837f16f54394d4c0d5ebcafef5c38b61aa628e58395fc758" exitCode=0
Nov 01 08:32:06.931779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.931795    4314 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="00e24beff5c4d4c81ccdb588bccd8012ebbca407505a543d2ea11be87993c286" exitCode=0
Nov 01 08:32:06.932058 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.931799    4314 generic.go:356] "Generic (PLEG): container finished" podID="65196622-05c5-4993-be70-8921dfd83293" containerID="28f42d501d5cfbfa6c70ebf5699ab5cda45061812ec112079dc19f21ecdc7529" exitCode=0
Nov 01 08:32:06.948332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.948306    4314 generic.go:356] "Generic (PLEG): container finished" podID="797457622f1c6f5ba3270c9efea90468" containerID="5b52598feee215ca54e9e7379a81a41dc2b6ddb51d9f07cd95f6c439e055783f" exitCode=0
Nov 01 08:32:06.952299 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.952284    4314 manager.go:324] Recovery completed
Nov 01 08:32:06.964455 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.964435    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:06.965412 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.965397    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:06.965486 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.965423    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:06.965486 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.965438    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:06.966627 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966614    4314 cpu_manager.go:226] "Starting" policy="none"
Nov 01 08:32:06.966627 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966625    4314 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 08:32:06.966718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966632    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:06.966718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966639    4314 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 08:32:06.966849 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966835    4314 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 08:32:06.966878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966849    4314 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 08:32:06.966878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966866    4314 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 08:32:06.966878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966874    4314 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 08:32:06.966957 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966882    4314 policy_none.go:50] "Start"
Nov 01 08:32:06.966957 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966895    4314 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 08:32:06.966957 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.966909    4314 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 08:32:06.967094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.967080    4314 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 08:32:06.967094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.967097    4314 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 08:32:06.967160 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.967106    4314 policy_none.go:44] "Start"
Nov 01 08:32:06.967412 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.967397    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:06.967472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.967425    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:06.967472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.967434    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:06.970892 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.970865    4314 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 08:32:06.975345 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975323    4314 manager.go:341] "Starting Device Plugin manager"
Nov 01 08:32:06.975345 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:06.975346    4314 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 08:32:06.975547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975357    4314 server.go:87] "Starting device plugin registration server"
Nov 01 08:32:06.975637 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975624    4314 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 08:32:06.975691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975637    4314 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 08:32:06.975739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975720    4314 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 08:32:06.975918 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975882    4314 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 08:32:06.975997 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975949    4314 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 08:32:06.975997 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.975958    4314 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 08:32:06.976094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.976020    4314 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T08:32:06.975915555Z","UUID":"87a1ac0e-989e-4ae7-ae53-260f734ab6c0","Handler":null,"Name":"","Endpoint":""}
Nov 01 08:32:06.977060 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:06.977040    4314 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 08:32:06.977204 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:06.977189    4314 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-93.us-west-2.compute.internal\" not found"
Nov 01 08:32:06.977282 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.977249    4314 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 08:32:06.977378 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:06.977362    4314 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 08:32:07.021653 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.021612    4314 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"]
Nov 01 08:32:07.021795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.021747    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:07.022629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.022612    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:07.022723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.022643    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:07.022723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.022657    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:07.022929 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.022916    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:07.023056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023041    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:07.023671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023648    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:07.023768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023680    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:07.023768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023691    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:07.023768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023658    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:07.023768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023744    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:07.023768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023756    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:07.023988 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:07.023841    4314 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.024023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.023998    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"67982dcee5ed5a322620afe889b9e391","Type":"ContainerStarted","Data":"443152204e948875d2b0390f5afdfb422f75866d3238b42edfd2c3646710383b"}
Nov 01 08:32:07.024052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024030    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"67982dcee5ed5a322620afe889b9e391","Type":"ContainerStarted","Data":"b6ff9391e53d321acc88b90e6bb22c5fac3e8dd9cdd8adc58ba6d0e0b6335d62"}
Nov 01 08:32:07.024079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024062    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"797457622f1c6f5ba3270c9efea90468","Type":"ContainerStarted","Data":"4c22f652223c33f9f3f93966216b12a7b677e3548143724d171747a023caf838"}
Nov 01 08:32:07.024079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024076    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"797457622f1c6f5ba3270c9efea90468","Type":"ContainerDied","Data":"5b52598feee215ca54e9e7379a81a41dc2b6ddb51d9f07cd95f6c439e055783f"}
Nov 01 08:32:07.024134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024089    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal" event={"ID":"797457622f1c6f5ba3270c9efea90468","Type":"ContainerStarted","Data":"020db964b07954e74980f2a0e3c1fa63603225b91cf042566d556c3205c64e63"}
Nov 01 08:32:07.024134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024096    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:07.024737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024721    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:07.024788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024753    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:07.024788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.024763    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:07.024891 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:07.024879    4314 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-93.us-west-2.compute.internal\" not found" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.075935 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.075916    4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:07.076679 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.076664    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:07.076739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.076689    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:07.076739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.076699    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:07.076739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.076718    4314 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.078473 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.078453    4314 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:07.078592 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.078457    4314 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:07.078592 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.078560    4314 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 08:32:07.078833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.078819    4314 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:07.089298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.089283    4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.126021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.125957    4314 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:07.170810 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.170789    4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:32:07.170924 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:07.170841    4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.170924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.170857    4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.179569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.179545    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.179639 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.179572    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67982dcee5ed5a322620afe889b9e391-config\") pod \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"67982dcee5ed5a322620afe889b9e391\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.179639 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.179590    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.181448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.181431    4314 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.181502 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.181494    4314 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.182032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.182017    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeNotReady"
Nov 01 08:32:07.182106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.182045    4314 setters.go:546] "Node became not ready" node="ip-10-0-134-93.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T08:32:07Z","lastTransitionTime":"2026-11-01T08:32:07Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 08:32:07.219494 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.219471    4314 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-93.us-west-2.compute.internal" is owned by different node (oldNodeID="233a805b-30fc-45a8-a406-dd7df2604f47", newNodeID=""), cleaning up...
Nov 01 08:32:07.246816 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.246791    4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:32:07.246904 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:07.246833    4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.279904 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.279880    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.279975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.279908    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.279975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.279926    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67982dcee5ed5a322620afe889b9e391-config\") pod \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"67982dcee5ed5a322620afe889b9e391\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.279975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.279935    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.279975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.279959    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67982dcee5ed5a322620afe889b9e391-config\") pod \"kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"67982dcee5ed5a322620afe889b9e391\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.280095 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.279983    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/797457622f1c6f5ba3270c9efea90468-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" (UID: \"797457622f1c6f5ba3270c9efea90468\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.447947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.447881    4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-93.us-west-2.compute.internal" event="NodeReady"
Nov 01 08:32:07.448058 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.447958    4314 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 08:32:07.878245 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.878213    4314 apiserver.go:52] "Watching apiserver"
Nov 01 08:32:07.950940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.950914    4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:07.975427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.975401    4314 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 08:32:07.975831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.975812    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-4lxtb","kube-system/global-pull-secret-syncer-5cgcj","openshift-ovn-kubernetes/ovnkube-node-m9vpp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-8cngb","openshift-network-operator/iptables-alerter-bgtrj","kube-system/konnectivity-agent-slw5x","openshift-ingress-canary/ingress-canary-2bz2q","openshift-insights/periodic-gathering-5m778-tsfb9","openshift-multus/network-metrics-daemon-rkhtc","kube-system/kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-nw2n6","openshift-image-registry/node-ca-p6sk5","openshift-dns/node-resolver-sjnbq","openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz","openshift-multus/multus-92bsb","openshift-dns/dns-default-q9w7q","openshift-insights/insights-runtime-extractor-q4bc2"]
Nov 01 08:32:07.976891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.976868    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:07.978454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.978431    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:07.983163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.983142    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:07.983602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:07.983583    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.007604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.007585    4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:32:08.007674 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.007622    4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-93.us-west-2.compute.internal"
Nov 01 08:32:08.023996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.023977    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz"
Nov 01 08:32:08.024525 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.024507    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.025547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.025513    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 08:32:08.025704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.025686    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\""
Nov 01 08:32:08.025808 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.025730    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cdn26\""
Nov 01 08:32:08.025870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.025835    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48mlk\""
Nov 01 08:32:08.025922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.025877    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hnxt\""
Nov 01 08:32:08.026139 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026125    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xk5lv\""
Nov 01 08:32:08.026686 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026667    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 08:32:08.026686 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026684    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 08:32:08.026818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026672    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 08:32:08.026818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026718    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.026818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026748    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.026947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026921    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.026947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026924    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.027008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026950    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.027008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.026968    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.027075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027024    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 08:32:08.027130 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027114    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.027130 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027117    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 08:32:08.027254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027127    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 08:32:08.027254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027152    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.027254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027154    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.027254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.027200    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 08:32:08.046844 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.046826    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 08:32:08.047629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047610    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 08:32:08.047720 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047674    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d96l\""
Nov 01 08:32:08.047720 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047626    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 08:32:08.047839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047698    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\""
Nov 01 08:32:08.047839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047802    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24cps\""
Nov 01 08:32:08.047930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047836    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 08:32:08.047930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047849    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqp75\""
Nov 01 08:32:08.047930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047884    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q5m6c\""
Nov 01 08:32:08.047930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047918    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 08:32:08.048097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047936    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 08:32:08.048097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.047995    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mnx4s\""
Nov 01 08:32:08.048097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.048046    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxggb\""
Nov 01 08:32:08.048097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.048091    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9vpz\""
Nov 01 08:32:08.048200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.048152    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h4rfg\""
Nov 01 08:32:08.048242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.048207    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\""
Nov 01 08:32:08.084465 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084443    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-k8s-cni-cncf-io\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.084561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084477    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5fr\" (UniqueName: \"kubernetes.io/projected/3c33083a-5961-4854-a5af-8b1de96e6eec-kube-api-access-8b5fr\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.084561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084502    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.084681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084566    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.084681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084594    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-tuned\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.084681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084618    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-ovn\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.084681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084659    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9b827d2-1aac-4471-9665-21c2575f6d0c-hosts-file\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.084875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084697    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.084875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084729    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-socket-dir-parent\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.084875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084744    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-tuned\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.084875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084802    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-kubelet\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.084875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084829    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.084875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084854    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-systemd\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.085080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084888    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-tmp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.085080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084962    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083acc55-e6ba-4016-abad-67be0cc4af2c-tmp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.085080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.084971    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-slash\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.085080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085005    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.085080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085008    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.085080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085027    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085107    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysconfig\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085127    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqvp\" (UniqueName: \"kubernetes.io/projected/083acc55-e6ba-4016-abad-67be0cc4af2c-kube-api-access-bwqvp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085155    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085172    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085215    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085269    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6czd\" (UniqueName: \"kubernetes.io/projected/c3286913-2025-4768-8e0f-65467b292ccc-kube-api-access-s6czd\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:08.085330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085297    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ad972a-7efb-4755-bdae-e2087f130fb4-serviceca\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.085597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085392    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.085597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085443    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-device-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.085597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:08.085597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085504    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-cnibin\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.085597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085545    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25832b5-0080-46b6-a543-f60a7d4da56e-cert\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:32:08.085597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085575    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-socket-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.085868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085627    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt27c\" (UniqueName: \"kubernetes.io/projected/d25832b5-0080-46b6-a543-f60a7d4da56e-kube-api-access-pt27c\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:32:08.085868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085749    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.085868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085764    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25832b5-0080-46b6-a543-f60a7d4da56e-cert\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:32:08.085868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085797    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-var-lib-kubelet\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.085868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085821    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.085868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085853    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b827d2-1aac-4471-9665-21c2575f6d0c-tmp-dir\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.086023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085948    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62767\" (UniqueName: \"kubernetes.io/projected/47ad972a-7efb-4755-bdae-e2087f130fb4-kube-api-access-62767\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.086023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085969    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085974    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-multus\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.086023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.085988    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b827d2-1aac-4471-9665-21c2575f6d0c-tmp-dir\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.086023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086000    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-multus-certs\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.086023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086015    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-conf\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.086177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086046    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-log-socket\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086080    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bade5291-3729-42d3-92da-7fe9e224a4ad-konnectivity-ca\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:32:08.086177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086129    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-bin\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086171    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-script-lib\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086267    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-etc-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086294    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-node-log\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086312    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-registration-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.086368 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086321    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-script-lib\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086368 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086344    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0369242-85ce-442a-a759-cd0d52c7fdef-host-slash\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.086426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086367    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1343be02-b373-4afb-8a8d-4e2780066e18-config-volume\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.086426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086398    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-bin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.086478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086436    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-netns\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.086478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086469    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-modprobe-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.086550 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-run\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.086550 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086507    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-lib-modules\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.086605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086560    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-netd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086593    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cqz\" (UniqueName: \"kubernetes.io/projected/c9b827d2-1aac-4471-9665-21c2575f6d0c-kube-api-access-j6cqz\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.086662 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086626    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-system-cni-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.086696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086666    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-system-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.086696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086689    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-var-lib-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.086752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086706    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w555j\" (UniqueName: \"kubernetes.io/projected/1343be02-b373-4afb-8a8d-4e2780066e18-kube-api-access-w555j\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.086752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086743    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-conf-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.086818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086778    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-dbus\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.086818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086810    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs477\" (UniqueName: \"kubernetes.io/projected/55eaf339-45f2-4182-bcab-a01f489180ee-kube-api-access-hs477\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.086942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.086856    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfq5\" (UniqueName: \"kubernetes.io/projected/a7c7f909-3dd3-473a-aacd-356bce893dcb-kube-api-access-ztfq5\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.087028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087012    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ad972a-7efb-4755-bdae-e2087f130fb4-host\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.087063 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087037    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.087063 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087052    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-hostroot\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.087133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087069    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.087133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087115    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcht\" (UniqueName: \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.087226 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087208    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-kubernetes\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.087278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087250    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-kubelet\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.087331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087281    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343be02-b373-4afb-8a8d-4e2780066e18-tmp-dir\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.087331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087298    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3370a9e2-dba0-475a-a84e-3d0d1cfab280-original-pull-secret\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.087393 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087328    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bsq\" (UniqueName: \"kubernetes.io/projected/65196622-05c5-4993-be70-8921dfd83293-kube-api-access-n2bsq\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.087457 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087441    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqzx\" (UniqueName: \"kubernetes.io/projected/0d455ab1-07fd-4110-8a3a-64b8f1fe83e2-kube-api-access-dhqzx\") pod \"migrator-66df5d496b-ppgzz\" (UID: \"0d455ab1-07fd-4110-8a3a-64b8f1fe83e2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz"
Nov 01 08:32:08.087497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087471    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-host\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.087497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087491    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.087619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087470    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343be02-b373-4afb-8a8d-4e2780066e18-tmp-dir\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.087619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087564    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0369242-85ce-442a-a759-cd0d52c7fdef-iptables-alerter-script\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.087619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087602    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:08.087732 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087629    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.087732 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087655    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-daemon-config\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.087732 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087688    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-kubelet-config\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.087838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087734    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0369242-85ce-442a-a759-cd0d52c7fdef-iptables-alerter-script\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.087838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087747    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-systemd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.087838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087781    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-config\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.087838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087796    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-cni-binary-copy\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.087838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087811    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087828    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bade5291-3729-42d3-92da-7fe9e224a4ad-agent-certs\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087858    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-env-overrides\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087869    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65196622-05c5-4993-be70-8921dfd83293-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087875    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjbx\" (UniqueName: \"kubernetes.io/projected/e0369242-85ce-442a-a759-cd0d52c7fdef-kube-api-access-jmjbx\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087915    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-systemd-units\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087944    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-ovnkube-config\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.087940    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-etc-kubernetes\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088010    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-sys-fs\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088023    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-cni-binary-copy\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088032    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-sys\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.088041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088038    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7c7f909-3dd3-473a-aacd-356bce893dcb-env-overrides\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.088390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088069    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-netns\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.088390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088087    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.088390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088114    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-cnibin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.088390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.088137    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bade5291-3729-42d3-92da-7fe9e224a4ad-agent-certs\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:32:08.089675 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.089663    4314 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 08:32:08.129625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129569    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 08:32:08.129625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129611    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.129769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129577    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.129769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129573    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.129769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129619    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.129769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129641    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 08:32:08.129769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129684    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.129965 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.129845    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 08:32:08.130188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130165    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.130372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130354    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.130438 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130428    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 08:32:08.130585 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130573    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.130616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130585    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 08:32:08.130649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130619    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.130689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130654    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.130689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130660    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.130689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130678    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.130774 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130688    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.130774 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.130706    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 08:32:08.131606 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.131594    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 08:32:08.136243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.136222    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ad972a-7efb-4755-bdae-e2087f130fb4-serviceca\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.136372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.136359    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bade5291-3729-42d3-92da-7fe9e224a4ad-konnectivity-ca\") pod \"konnectivity-agent-slw5x\" (UID: \"bade5291-3729-42d3-92da-7fe9e224a4ad\") " pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:32:08.137302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.137287    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1343be02-b373-4afb-8a8d-4e2780066e18-config-volume\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.138416 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.138402    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-daemon-config\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.139636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.139621    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5fr\" (UniqueName: \"kubernetes.io/projected/3c33083a-5961-4854-a5af-8b1de96e6eec-kube-api-access-8b5fr\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.140033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.140018    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqvp\" (UniqueName: \"kubernetes.io/projected/083acc55-e6ba-4016-abad-67be0cc4af2c-kube-api-access-bwqvp\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.140073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.140030    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6czd\" (UniqueName: \"kubernetes.io/projected/c3286913-2025-4768-8e0f-65467b292ccc-kube-api-access-s6czd\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:08.140646 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.140631    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62767\" (UniqueName: \"kubernetes.io/projected/47ad972a-7efb-4755-bdae-e2087f130fb4-kube-api-access-62767\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.140797 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.140783    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt27c\" (UniqueName: \"kubernetes.io/projected/d25832b5-0080-46b6-a543-f60a7d4da56e-kube-api-access-pt27c\") pod \"ingress-canary-2bz2q\" (UID: \"d25832b5-0080-46b6-a543-f60a7d4da56e\") " pod="openshift-ingress-canary/ingress-canary-2bz2q"
Nov 01 08:32:08.141377 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.141356    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cqz\" (UniqueName: \"kubernetes.io/projected/c9b827d2-1aac-4471-9665-21c2575f6d0c-kube-api-access-j6cqz\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.141686 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.141671    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w555j\" (UniqueName: \"kubernetes.io/projected/1343be02-b373-4afb-8a8d-4e2780066e18-kube-api-access-w555j\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.141846 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.141832    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfq5\" (UniqueName: \"kubernetes.io/projected/a7c7f909-3dd3-473a-aacd-356bce893dcb-kube-api-access-ztfq5\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.141887 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.141861    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs477\" (UniqueName: \"kubernetes.io/projected/55eaf339-45f2-4182-bcab-a01f489180ee-kube-api-access-hs477\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.142046 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.142032    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bsq\" (UniqueName: \"kubernetes.io/projected/65196622-05c5-4993-be70-8921dfd83293-kube-api-access-n2bsq\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.142554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.142519    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjbx\" (UniqueName: \"kubernetes.io/projected/e0369242-85ce-442a-a759-cd0d52c7fdef-kube-api-access-jmjbx\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.147790 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.147775    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 08:32:08.147867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.147853    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmhw9\""
Nov 01 08:32:08.147900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.147890    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8rfsj\""
Nov 01 08:32:08.188521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188495    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-systemd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.188628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188525    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.188628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188554    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-systemd-units\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.188628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188575    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-etc-kubernetes\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.188628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188597    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-sys-fs\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.188628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188607    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-systemd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188635    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-systemd-units\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188649    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-etc-kubernetes\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188653    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-sys-fs\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188672    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-sys\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188695    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188712    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-netns\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188727    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188731    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-sys\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188741    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-cnibin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188763    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-k8s-cni-cncf-io\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188771    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-netns\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.188798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188786    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188807    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-k8s-cni-cncf-io\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188808    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188829    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-cnibin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188842    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-ovn\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.188858    4314 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188875    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-ovn\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188878    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188876    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9b827d2-1aac-4471-9665-21c2575f6d0c-hosts-file\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188914    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-socket-dir-parent\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188917    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9b827d2-1aac-4471-9665-21c2575f6d0c-hosts-file\") pod \"node-resolver-sjnbq\" (UID: \"c9b827d2-1aac-4471-9665-21c2575f6d0c\") " pod="openshift-dns/node-resolver-sjnbq"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188926    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.188931    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:08.688910074 +0000 UTC m=+1.883304611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188942    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-socket-dir-parent\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188972    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-kubelet\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.188995    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189011    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-systemd\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189026    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-slash\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189035    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-kubelet\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189051    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189041    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189071    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-systemd\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189084    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysconfig\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189101    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-slash\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189110    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189147    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-run-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.189148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189150    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysconfig\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189171    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189199    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189225    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-device-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189233    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189263    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189275    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-device-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189284    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189291    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189327    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-cnibin\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189358    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189358    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-socket-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.189388    4314 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189392    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189405    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-socket-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189408    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-var-lib-kubelet\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.189437    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:32:08.689421296 +0000 UTC m=+1.883815834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : secret "metrics-daemon-secret" not found
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189444    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-var-lib-kubelet\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189452    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-cnibin\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189473    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-multus\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189490    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189498    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-multus-certs\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189520    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-conf\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189558    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-log-socket\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189563    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-multus\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189582    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-bin\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189592    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-multus-certs\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189609    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-etc-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189640    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-bin\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189645    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-node-log\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189656    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-etc-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189663    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-log-socket\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189677    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-sysctl-conf\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189678    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-registration-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189699    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-node-log\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189706    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0369242-85ce-442a-a759-cd0d52c7fdef-host-slash\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189714    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189731    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-bin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189755    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-netns\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189770    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0369242-85ce-442a-a759-cd0d52c7fdef-host-slash\") pod \"iptables-alerter-bgtrj\" (UID: \"e0369242-85ce-442a-a759-cd0d52c7fdef\") " pod="openshift-network-operator/iptables-alerter-bgtrj"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189778    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-modprobe-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189772    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55eaf339-45f2-4182-bcab-a01f489180ee-registration-dir\") pod \"aws-ebs-csi-driver-node-cv9tg\" (UID: \"55eaf339-45f2-4182-bcab-a01f489180ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cv9tg"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189789    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-run-netns\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189775    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-host-var-lib-cni-bin\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189804    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-run\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189829    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-lib-modules\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189844    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-run\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189852    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007b6d34-3e90-4550-a5e8-508825a98796-data-volume\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189861    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-modprobe-d\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189874    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-netd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189906    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-system-cni-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189926    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-cni-netd\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189909    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/007b6d34-3e90-4550-a5e8-508825a98796-data-volume\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189929    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-lib-modules\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189931    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-system-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189954    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-system-cni-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189962    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwn58\" (UniqueName: \"kubernetes.io/projected/007b6d34-3e90-4550-a5e8-508825a98796-kube-api-access-jwn58\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189969    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-system-cni-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.189986    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-var-lib-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190002    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-conf-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190018    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-dbus\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190065    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-dbus\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190065    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ad972a-7efb-4755-bdae-e2087f130fb4-host\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190095    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ad972a-7efb-4755-bdae-e2087f130fb4-host\") pod \"node-ca-p6sk5\" (UID: \"47ad972a-7efb-4755-bdae-e2087f130fb4\") " pod="openshift-image-registry/node-ca-p6sk5"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190099    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.190075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190111    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-var-lib-openvswitch\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190127    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-hostroot\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190137    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-multus-conf-dir\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190141    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65196622-05c5-4993-be70-8921dfd83293-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cngb\" (UID: \"65196622-05c5-4993-be70-8921dfd83293\") " pod="openshift-multus/multus-additional-cni-plugins-8cngb"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190152    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcht\" (UniqueName: \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190159    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c33083a-5961-4854-a5af-8b1de96e6eec-hostroot\") pod \"multus-92bsb\" (UID: \"3c33083a-5961-4854-a5af-8b1de96e6eec\") " pod="openshift-multus/multus-92bsb"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190186    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-kubernetes\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190230    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-etc-kubernetes\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190248    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-kubelet\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190266    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqzx\" (UniqueName: \"kubernetes.io/projected/0d455ab1-07fd-4110-8a3a-64b8f1fe83e2-kube-api-access-dhqzx\") pod \"migrator-66df5d496b-ppgzz\" (UID: \"0d455ab1-07fd-4110-8a3a-64b8f1fe83e2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190280    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-host\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190296    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007b6d34-3e90-4550-a5e8-508825a98796-crio-socket\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190312    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190330    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190362    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-kubelet-config\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190370    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/083acc55-e6ba-4016-abad-67be0cc4af2c-host\") pod \"tuned-nw2n6\" (UID: \"083acc55-e6ba-4016-abad-67be0cc4af2c\") " pod="openshift-cluster-node-tuning-operator/tuned-nw2n6"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190373    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190382    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7c7f909-3dd3-473a-aacd-356bce893dcb-host-kubelet\") pod \"ovnkube-node-m9vpp\" (UID: \"a7c7f909-3dd3-473a-aacd-356bce893dcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:08.191831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.190426    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3370a9e2-dba0-475a-a84e-3d0d1cfab280-kubelet-config\") pod \"global-pull-secret-syncer-5cgcj\" (UID: \"3370a9e2-dba0-475a-a84e-3d0d1cfab280\") " pod="kube-system/global-pull-secret-syncer-5cgcj"
Nov 01 08:32:08.247314 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.247281    4314 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 08:32:08.249629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.249603    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcht\" (UniqueName: \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\") pod \"periodic-gathering-5m778-tsfb9\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") " pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.249728 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.249664    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjlk\" (UniqueName: \"kubernetes.io/projected/6483c164-427b-49ee-8a3c-bf576168ed77-kube-api-access-mgjlk\") pod \"network-check-target-4lxtb\" (UID: \"6483c164-427b-49ee-8a3c-bf576168ed77\") " pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:08.249949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.249933    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqzx\" (UniqueName: \"kubernetes.io/projected/0d455ab1-07fd-4110-8a3a-64b8f1fe83e2-kube-api-access-dhqzx\") pod \"migrator-66df5d496b-ppgzz\" (UID: \"0d455ab1-07fd-4110-8a3a-64b8f1fe83e2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz"
Nov 01 08:32:08.277163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.277147    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:08.284287 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.284268    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:32:08.290577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.290560    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwn58\" (UniqueName: \"kubernetes.io/projected/007b6d34-3e90-4550-a5e8-508825a98796-kube-api-access-jwn58\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.290629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.290591    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007b6d34-3e90-4550-a5e8-508825a98796-crio-socket\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.290629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.290619    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.290705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.290667    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.290737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.290671    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/007b6d34-3e90-4550-a5e8-508825a98796-crio-socket\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.290737 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.290707    4314 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:08.290798 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.290749    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:08.790737853 +0000 UTC m=+1.985132410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:08.290841 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.290799    4314 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:08.290841 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.290825    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:08.790818684 +0000 UTC m=+1.985213216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:08.311375 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.311350    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vxfb4"]
Nov 01 08:32:08.340092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.340074    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.347077 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.347042    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz"
Nov 01 08:32:08.348569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.348545    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwn58\" (UniqueName: \"kubernetes.io/projected/007b6d34-3e90-4550-a5e8-508825a98796-kube-api-access-jwn58\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.369689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.369661    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vxfb4"]
Nov 01 08:32:08.389830 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.389806    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 08:32:08.389932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.389918    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 08:32:08.390037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.390023    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 08:32:08.390108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.390097    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 08:32:08.390214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.390203    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hr7ht\""
Nov 01 08:32:08.391075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.391052    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-signing-key\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.391263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.391233    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffn2\" (UniqueName: \"kubernetes.io/projected/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-kube-api-access-rffn2\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.391324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.391303    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-signing-cabundle\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.492102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.492073    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffn2\" (UniqueName: \"kubernetes.io/projected/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-kube-api-access-rffn2\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.492235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.492111    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-signing-cabundle\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.492235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.492150    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-signing-key\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.492868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.492850    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-signing-cabundle\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.494293 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.494267    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-signing-key\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.549544 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.549513    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffn2\" (UniqueName: \"kubernetes.io/projected/a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d-kube-api-access-rffn2\") pod \"service-ca-5fffc97bdf-vxfb4\" (UID: \"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.591635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.591600    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:08.668251 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.667368    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4lxtb"]
Nov 01 08:32:08.672323 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:08.672284    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6483c164_427b_49ee_8a3c_bf576168ed77.slice/crio-67aadefe4d5cbd1ee6802abe0b83793997f0badaf6b1e1ad06ff2ad3616e2ff6 WatchSource:0}: Error finding container 67aadefe4d5cbd1ee6802abe0b83793997f0badaf6b1e1ad06ff2ad3616e2ff6: Status 404 returned error can't find the container with id 67aadefe4d5cbd1ee6802abe0b83793997f0badaf6b1e1ad06ff2ad3616e2ff6
Nov 01 08:32:08.674443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.674429    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:32:08.680348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.680331    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4"
Nov 01 08:32:08.692901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.692874    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:08.692987 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.692940    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:08.693023 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.692974    4314 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:08.693053 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.693035    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:32:09.693022052 +0000 UTC m=+2.887416600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : secret "metrics-daemon-secret" not found
Nov 01 08:32:08.693053 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.693038    4314 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:08.693113 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.693087    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:09.693076221 +0000 UTC m=+2.887470754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:08.716749 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.716727    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5m778-tsfb9"]
Nov 01 08:32:08.719697 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:08.719673    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda069a9b7_205f_4e32_8b62_0a9a1e67d59b.slice/crio-03f9d9758c56585ff23765b80a9b3b4f74b5f238b2d3b7efae45e6c2e3e859a9 WatchSource:0}: Error finding container 03f9d9758c56585ff23765b80a9b3b4f74b5f238b2d3b7efae45e6c2e3e859a9: Status 404 returned error can't find the container with id 03f9d9758c56585ff23765b80a9b3b4f74b5f238b2d3b7efae45e6c2e3e859a9
Nov 01 08:32:08.750286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.750255    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:08.755487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.755465    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz"]
Nov 01 08:32:08.758874 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:08.758848    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d455ab1_07fd_4110_8a3a_64b8f1fe83e2.slice/crio-92859cf45c42c50b86e78ad7dbc00a2e97e85da02f5e923015871ec8b824eb09 WatchSource:0}: Error finding container 92859cf45c42c50b86e78ad7dbc00a2e97e85da02f5e923015871ec8b824eb09: Status 404 returned error can't find the container with id 92859cf45c42c50b86e78ad7dbc00a2e97e85da02f5e923015871ec8b824eb09
Nov 01 08:32:08.793898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.793875    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.793973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.793929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:08.794017 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.794004    4314 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:08.794062 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.794031    4314 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:08.794092 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.794073    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:09.794059113 +0000 UTC m=+2.988453661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:08.794092 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:08.794086    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:09.794081032 +0000 UTC m=+2.988475565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:08.954222 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.954158    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m778-tsfb9" event={"ID":"a069a9b7-205f-4e32-8b62-0a9a1e67d59b","Type":"ContainerStarted","Data":"03f9d9758c56585ff23765b80a9b3b4f74b5f238b2d3b7efae45e6c2e3e859a9"}
Nov 01 08:32:08.955246 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.955224    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz" event={"ID":"0d455ab1-07fd-4110-8a3a-64b8f1fe83e2","Type":"ContainerStarted","Data":"92859cf45c42c50b86e78ad7dbc00a2e97e85da02f5e923015871ec8b824eb09"}
Nov 01 08:32:08.956245 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:08.956219    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4lxtb" event={"ID":"6483c164-427b-49ee-8a3c-bf576168ed77","Type":"ContainerStarted","Data":"67aadefe4d5cbd1ee6802abe0b83793997f0badaf6b1e1ad06ff2ad3616e2ff6"}
Nov 01 08:32:09.056438 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.056413    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vxfb4"]
Nov 01 08:32:09.059865 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:09.059835    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a22fa9_ba10_4046_b3fb_8cb3df9f5f4d.slice/crio-d87f88ff1600bc7f6f3c2cc16c3bf6a215a37c19d065697991b3b5d195515a5c WatchSource:0}: Error finding container d87f88ff1600bc7f6f3c2cc16c3bf6a215a37c19d065697991b3b5d195515a5c: Status 404 returned error can't find the container with id d87f88ff1600bc7f6f3c2cc16c3bf6a215a37c19d065697991b3b5d195515a5c
Nov 01 08:32:09.213102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.213032    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:09.698741 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.698713    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:09.698846 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.698763    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:09.698894 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.698856    4314 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:09.698894 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.698857    4314 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:09.698953 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.698912    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.698896675 +0000 UTC m=+4.893291228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : secret "metrics-daemon-secret" not found
Nov 01 08:32:09.698953 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.698925    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.698919235 +0000 UTC m=+4.893313768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:09.799200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.799172    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:09.799347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.799220    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:09.799347 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.799313    4314 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:09.799408 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.799348    4314 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:09.799408 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.799396    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.799376446 +0000 UTC m=+4.993770999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:09.799473 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:09.799416    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.79940725 +0000 UTC m=+4.993801783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:09.961283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:09.961129    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4" event={"ID":"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d","Type":"ContainerStarted","Data":"d87f88ff1600bc7f6f3c2cc16c3bf6a215a37c19d065697991b3b5d195515a5c"}
Nov 01 08:32:11.441228 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.441176    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:11.594045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.594012    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:11.714883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.714790    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:11.715054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.714901    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:11.715054 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:11.714924    4314 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:11.715054 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:11.715004    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs podName:c3286913-2025-4768-8e0f-65467b292ccc nodeName:}" failed. No retries permitted until 2026-11-01 08:32:15.714982201 +0000 UTC m=+8.909376734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs") pod "network-metrics-daemon-rkhtc" (UID: "c3286913-2025-4768-8e0f-65467b292ccc") : secret "metrics-daemon-secret" not found
Nov 01 08:32:11.715054 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:11.715046    4314 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:11.715198 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:11.715102    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:15.715086985 +0000 UTC m=+8.909481520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:11.816137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.816097    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:11.816317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.816207    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:11.816317 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:11.816269    4314 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:11.816442 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:11.816333    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:15.816317761 +0000 UTC m=+9.010712293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:11.816855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.816833    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/007b6d34-3e90-4550-a5e8-508825a98796-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:11.969387 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.969304    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz" event={"ID":"0d455ab1-07fd-4110-8a3a-64b8f1fe83e2","Type":"ContainerStarted","Data":"22bec730fc909d629ec0d51439724826f7692c72de540ba91fe9f84ecf53d6c7"}
Nov 01 08:32:11.971247 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:11.971214    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m778-tsfb9" event={"ID":"a069a9b7-205f-4e32-8b62-0a9a1e67d59b","Type":"ContainerStarted","Data":"8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9"}
Nov 01 08:32:12.189648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:12.189586    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ppgzz" podStartSLOduration=4.023489426 podStartE2EDuration="6.189570906s" totalImagesPullingTime="2.16608148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:12.080872929 +0000 UTC m=+5.275267485" watchObservedRunningTime="2026-11-01 08:32:12.189570906 +0000 UTC m=+5.383965455"
Nov 01 08:32:12.974588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:12.974485    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4" event={"ID":"a3a22fa9-ba10-4046-b3fb-8cb3df9f5f4d","Type":"ContainerStarted","Data":"596af56405e632cace2f8e340252afe1d02b53df1836cd257ea6990e2a5e3d68"}
Nov 01 08:32:12.975974 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:12.975943    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4lxtb" event={"ID":"6483c164-427b-49ee-8a3c-bf576168ed77","Type":"ContainerStarted","Data":"5ee0e452282824d3d7f0b8f7e46b1ceaeb9d2995fa6e49157c02dd05a3ed6db0"}
Nov 01 08:32:12.976238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:12.976210    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:13.081358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:13.081315    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-vxfb4" podStartSLOduration=1.831035728 podStartE2EDuration="5.081304061s" totalImagesPullingTime="3.250268333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:13.080754918 +0000 UTC m=+6.275149550" watchObservedRunningTime="2026-11-01 08:32:13.081304061 +0000 UTC m=+6.275698670"
Nov 01 08:32:13.516238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:13.516199    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:13.883572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:13.883493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:14.091880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:14.091847    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:14.237146 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:14.237116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:14.940520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:14.940493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:15.176143 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.176110    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:15.323421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.323385    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:15.471974 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.471939    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:15.588504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.588433    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sjnbq_c9b827d2-1aac-4471-9665-21c2575f6d0c/dns-node-resolver/0.log"
Nov 01 08:32:15.661292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.661258    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:15.743594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.743550    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:15.743784 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.743628    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:15.743784 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:15.743639    4314 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:15.743784 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:15.743694    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls podName:1343be02-b373-4afb-8a8d-4e2780066e18 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:23.743681462 +0000 UTC m=+16.938075995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls") pod "dns-default-q9w7q" (UID: "1343be02-b373-4afb-8a8d-4e2780066e18") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:15.746138 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.746112    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3286913-2025-4768-8e0f-65467b292ccc-metrics-certs\") pod \"network-metrics-daemon-rkhtc\" (UID: \"c3286913-2025-4768-8e0f-65467b292ccc\") " pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:15.778968 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.778938    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhtc"
Nov 01 08:32:15.811074 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.811041    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:15.850600 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:15.849859    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:15.850600 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:15.850041    4314 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:15.850600 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:15.850105    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls podName:007b6d34-3e90-4550-a5e8-508825a98796 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:23.850086504 +0000 UTC m=+17.044481045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4bc2" (UID: "007b6d34-3e90-4550-a5e8-508825a98796") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:16.014307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.014280    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:16.145270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.145196    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkhtc"]
Nov 01 08:32:16.148865 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:16.148826    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3286913_2025_4768_8e0f_65467b292ccc.slice/crio-20d1918b24567e2fd27d7a4f523cab041f2339984e2de1596ef57266a4e1f30f WatchSource:0}: Error finding container 20d1918b24567e2fd27d7a4f523cab041f2339984e2de1596ef57266a4e1f30f: Status 404 returned error can't find the container with id 20d1918b24567e2fd27d7a4f523cab041f2339984e2de1596ef57266a4e1f30f
Nov 01 08:32:16.182769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.182735    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:16.327791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.327752    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:16.356343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.356318    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:16.388868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.388844    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6sk5_47ad972a-7efb-4755-bdae-e2087f130fb4/node-ca/0.log"
Nov 01 08:32:16.986488 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.986448    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkhtc" event={"ID":"c3286913-2025-4768-8e0f-65467b292ccc","Type":"ContainerStarted","Data":"20d1918b24567e2fd27d7a4f523cab041f2339984e2de1596ef57266a4e1f30f"}
Nov 01 08:32:16.989040 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:16.989011    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2bz2q_d25832b5-0080-46b6-a543-f60a7d4da56e/serve-healthcheck-canary/0.log"
Nov 01 08:32:17.389170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.389090    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ppgzz_0d455ab1-07fd-4110-8a3a-64b8f1fe83e2/migrator/0.log"
Nov 01 08:32:17.417561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.417514    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:17.452369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.452346    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:17.452594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.452579    4314 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 08:32:17.614094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.614065    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:17.812272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.812246    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:17.992234 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.992198    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkhtc" event={"ID":"c3286913-2025-4768-8e0f-65467b292ccc","Type":"ContainerStarted","Data":"f86ea7262e1d12b25dceca01e623af964100ec9c8db0a72601043cb92df9bbe6"}
Nov 01 08:32:17.992394 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:17.992258    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkhtc" event={"ID":"c3286913-2025-4768-8e0f-65467b292ccc","Type":"ContainerStarted","Data":"94bd26a1d0bc8eae05c9ecc3d5ae2438ee2bba41de39f1ed9f58f697d8731e00"}
Nov 01 08:32:18.018760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:18.018708    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:18.217324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:18.217288    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:19.414649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:19.414617    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:19.617475 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:19.617435    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:19.815588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:19.815556    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:20.016306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:20.016269    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:20.219159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:20.219077    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:21.220911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:21.220882    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:21.613001 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:21.612970    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:21.814009 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:21.813980    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:21.863660 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:21.863600    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-slw5x"
Nov 01 08:32:22.018263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:22.018234    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:22.417482 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:22.417450    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:23.214697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.214666    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:23.445142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.445106    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:23.591269 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.591229    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:23.738308 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.738280    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:23.805372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.805342    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:23.807457 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.807437    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1343be02-b373-4afb-8a8d-4e2780066e18-metrics-tls\") pod \"dns-default-q9w7q\" (UID: \"1343be02-b373-4afb-8a8d-4e2780066e18\") " pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:23.883686 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.883621    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:23.892632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.892606    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:23.906313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.906291    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:23.908568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.908515    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/007b6d34-3e90-4550-a5e8-508825a98796-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4bc2\" (UID: \"007b6d34-3e90-4550-a5e8-508825a98796\") " pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:23.925574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:23.925550    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4bc2"
Nov 01 08:32:24.037516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:24.037488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:24.185009 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:24.184948    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:24.245456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:24.245424    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q9w7q"]
Nov 01 08:32:24.248723 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:24.248696    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1343be02_b373_4afb_8a8d_4e2780066e18.slice/crio-f5bd60ccc3cc3886b0cd265125d239ef97b21006040116d4d4fd17cab7819795 WatchSource:0}: Error finding container f5bd60ccc3cc3886b0cd265125d239ef97b21006040116d4d4fd17cab7819795: Status 404 returned error can't find the container with id f5bd60ccc3cc3886b0cd265125d239ef97b21006040116d4d4fd17cab7819795
Nov 01 08:32:24.291698 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:24.291670    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4bc2"]
Nov 01 08:32:24.294862 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:24.294843    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007b6d34_3e90_4550_a5e8_508825a98796.slice/crio-256f95950bffd9cc961ccf47cf9aaf6f63b986483c76ea48246ba5dab6262e6f WatchSource:0}: Error finding container 256f95950bffd9cc961ccf47cf9aaf6f63b986483c76ea48246ba5dab6262e6f: Status 404 returned error can't find the container with id 256f95950bffd9cc961ccf47cf9aaf6f63b986483c76ea48246ba5dab6262e6f
Nov 01 08:32:24.333564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:24.333545    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:24.616197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:24.616164    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:25.013269 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:25.013239    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q9w7q" event={"ID":"1343be02-b373-4afb-8a8d-4e2780066e18","Type":"ContainerStarted","Data":"f5bd60ccc3cc3886b0cd265125d239ef97b21006040116d4d4fd17cab7819795"}
Nov 01 08:32:25.014594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:25.014572    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4bc2" event={"ID":"007b6d34-3e90-4550-a5e8-508825a98796","Type":"ContainerStarted","Data":"3046786da0c453faaf7a3919bd5a13aa69cf6d90633a1c2a71e04fa6c8f3d666"}
Nov 01 08:32:25.014706 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:25.014614    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4bc2" event={"ID":"007b6d34-3e90-4550-a5e8-508825a98796","Type":"ContainerStarted","Data":"256f95950bffd9cc961ccf47cf9aaf6f63b986483c76ea48246ba5dab6262e6f"}
Nov 01 08:32:26.020212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:26.020175    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4bc2" event={"ID":"007b6d34-3e90-4550-a5e8-508825a98796","Type":"ContainerStarted","Data":"723380fbfb04eecb5ff7d32aaffdb1e27136a17aae0e17172ee7ed38f1411e93"}
Nov 01 08:32:26.886420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:26.886388    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:27.034067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:27.034035    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q9w7q" event={"ID":"1343be02-b373-4afb-8a8d-4e2780066e18","Type":"ContainerStarted","Data":"4c7f35d2ddbfb809685cbab536e094ae29cc0a649682f2a22675dc823e8ddae1"}
Nov 01 08:32:27.034565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:27.034119    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q9w7q" event={"ID":"1343be02-b373-4afb-8a8d-4e2780066e18","Type":"ContainerStarted","Data":"4e1f42c0dc5e539f5fda4e202ce0142e956a601602736b9cb59738397361d59d"}
Nov 01 08:32:27.034565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:27.034168    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:27.034565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:27.034222    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:28.039396 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:28.039358    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4bc2" event={"ID":"007b6d34-3e90-4550-a5e8-508825a98796","Type":"ContainerStarted","Data":"8414f8fb26a354c0e7ba030f3fa3efdfde8642d032d6eb693cebd9b6ac3af94a"}
Nov 01 08:32:28.878868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:28.878835    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:29.044592 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.044558    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:29.198193 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.198118    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-sz5pz"]
Nov 01 08:32:29.201100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.201084    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.241092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.241065    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/daddcc0b-25a9-4a95-af81-5bdf8e63a4b9-tmp\") pod \"downloads-6dd789c8b7-sz5pz\" (UID: \"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9\") " pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.241216 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.241113    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59hr\" (UniqueName: \"kubernetes.io/projected/daddcc0b-25a9-4a95-af81-5bdf8e63a4b9-kube-api-access-p59hr\") pod \"downloads-6dd789c8b7-sz5pz\" (UID: \"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9\") " pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.249233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.249208    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 08:32:29.249233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.249214    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 08:32:29.249435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.249421    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-p2cr7\""
Nov 01 08:32:29.296002 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.295983    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sz5pz"]
Nov 01 08:32:29.296403 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.296386    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d4989f77b-l5wws"]
Nov 01 08:32:29.299217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.299204    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.334046 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.334011    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d4989f77b-l5wws"]
Nov 01 08:32:29.342371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342350    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59hr\" (UniqueName: \"kubernetes.io/projected/daddcc0b-25a9-4a95-af81-5bdf8e63a4b9-kube-api-access-p59hr\") pod \"downloads-6dd789c8b7-sz5pz\" (UID: \"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9\") " pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.342472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342388    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b45cc43b-ce41-443f-944c-699e3236d54e-trusted-ca\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342419    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b45cc43b-ce41-443f-944c-699e3236d54e-installation-pull-secrets\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342582 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342511    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b45cc43b-ce41-443f-944c-699e3236d54e-image-registry-private-configuration\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342618 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342595    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-registry-tls\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342619    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b45cc43b-ce41-443f-944c-699e3236d54e-ca-trust-extracted\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342647    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b45cc43b-ce41-443f-944c-699e3236d54e-registry-certificates\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342666    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-bound-sa-token\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342683    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dkg\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-kube-api-access-k8dkg\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.342780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.342763    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/daddcc0b-25a9-4a95-af81-5bdf8e63a4b9-tmp\") pod \"downloads-6dd789c8b7-sz5pz\" (UID: \"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9\") " pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.343071 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.343056    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/daddcc0b-25a9-4a95-af81-5bdf8e63a4b9-tmp\") pod \"downloads-6dd789c8b7-sz5pz\" (UID: \"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9\") " pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.394051 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.394024    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 08:32:29.394178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.394055    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 08:32:29.394178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.394057    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pn77d\""
Nov 01 08:32:29.394178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.394096    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 08:32:29.394285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.394262    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 08:32:29.400598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.400566    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:29.401550 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.401517    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59hr\" (UniqueName: \"kubernetes.io/projected/daddcc0b-25a9-4a95-af81-5bdf8e63a4b9-kube-api-access-p59hr\") pod \"downloads-6dd789c8b7-sz5pz\" (UID: \"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9\") " pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.443656 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443633    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b45cc43b-ce41-443f-944c-699e3236d54e-image-registry-private-configuration\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.443780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443663    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-registry-tls\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.443780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443682    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b45cc43b-ce41-443f-944c-699e3236d54e-ca-trust-extracted\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.443780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443704    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b45cc43b-ce41-443f-944c-699e3236d54e-registry-certificates\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.443780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443734    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-bound-sa-token\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.443780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443761    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dkg\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-kube-api-access-k8dkg\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.444042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443825    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b45cc43b-ce41-443f-944c-699e3236d54e-trusted-ca\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.444042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.443855    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b45cc43b-ce41-443f-944c-699e3236d54e-installation-pull-secrets\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.444174 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.444149    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b45cc43b-ce41-443f-944c-699e3236d54e-ca-trust-extracted\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.444657 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.444641    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b45cc43b-ce41-443f-944c-699e3236d54e-registry-certificates\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.445036 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.445014    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b45cc43b-ce41-443f-944c-699e3236d54e-trusted-ca\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.446195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.446174    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b45cc43b-ce41-443f-944c-699e3236d54e-image-registry-private-configuration\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.446598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.446582    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-registry-tls\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.446712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.446692    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b45cc43b-ce41-443f-944c-699e3236d54e-installation-pull-secrets\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.500287 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.500251    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-bound-sa-token\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.500573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.500526    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dkg\" (UniqueName: \"kubernetes.io/projected/b45cc43b-ce41-443f-944c-699e3236d54e-kube-api-access-k8dkg\") pod \"image-registry-6d4989f77b-l5wws\" (UID: \"b45cc43b-ce41-443f-944c-699e3236d54e\") " pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.512218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.512200    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:29.546479 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.546431    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:29.609853 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.609822    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:29.809521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.809445    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:29.916385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.916351    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sz5pz"]
Nov 01 08:32:29.919977 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:29.919944    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaddcc0b_25a9_4a95_af81_5bdf8e63a4b9.slice/crio-c9e297613a0e97453d6f0c6df01ab3ba35e91ae77a3b1a08a217a4d4ca2da2de WatchSource:0}: Error finding container c9e297613a0e97453d6f0c6df01ab3ba35e91ae77a3b1a08a217a4d4ca2da2de: Status 404 returned error can't find the container with id c9e297613a0e97453d6f0c6df01ab3ba35e91ae77a3b1a08a217a4d4ca2da2de
Nov 01 08:32:29.953106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.953073    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:29.971498 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:29.971467    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d4989f77b-l5wws"]
Nov 01 08:32:29.974662 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:29.974637    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45cc43b_ce41_443f_944c_699e3236d54e.slice/crio-c20b2bb74e415fcb7e71b2af0ab95eb9e2ee4c4d91914d34f3779e4b5283ed2b WatchSource:0}: Error finding container c20b2bb74e415fcb7e71b2af0ab95eb9e2ee4c4d91914d34f3779e4b5283ed2b: Status 404 returned error can't find the container with id c20b2bb74e415fcb7e71b2af0ab95eb9e2ee4c4d91914d34f3779e4b5283ed2b
Nov 01 08:32:30.046472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:30.046442    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws" event={"ID":"b45cc43b-ce41-443f-944c-699e3236d54e","Type":"ContainerStarted","Data":"f6d0dbeb6fe90bf33e7a78c2c6783bfefac246c3f65f8c30d0c3adf383775750"}
Nov 01 08:32:30.046877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:30.046492    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws" event={"ID":"b45cc43b-ce41-443f-944c-699e3236d54e","Type":"ContainerStarted","Data":"c20b2bb74e415fcb7e71b2af0ab95eb9e2ee4c4d91914d34f3779e4b5283ed2b"}
Nov 01 08:32:30.046877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:30.046557    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:30.047987 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:30.047928    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sz5pz" event={"ID":"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9","Type":"ContainerStarted","Data":"c9e297613a0e97453d6f0c6df01ab3ba35e91ae77a3b1a08a217a4d4ca2da2de"}
Nov 01 08:32:30.162515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:30.162414    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws" podStartSLOduration=1.162400052 podStartE2EDuration="1.162400052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:30.161824572 +0000 UTC m=+23.356219164" watchObservedRunningTime="2026-11-01 08:32:30.162400052 +0000 UTC m=+23.356794608"
Nov 01 08:32:31.073326 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:31.073287    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:31.224897 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:31.224853    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:31.370743 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:31.370664    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:32.053465 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:32.053433    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:32.317520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:32.317395    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:33.087952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:33.087912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:33.232773 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:33.232724    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:37.041580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:37.041553    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q9w7q"
Nov 01 08:32:37.305252 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:37.305178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:37.408068 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:37.408043    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9vpp"
Nov 01 08:32:37.458922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:37.458897    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:37.599722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:37.599643    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:37.739817 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:37.739783    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:39.079958 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:39.079927    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:41.324037 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:32:41.323943    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:32:43.872909 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:43.872874    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:43.982864 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:43.982833    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4lxtb"
Nov 01 08:32:44.013142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:44.013106    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:44.909069 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:44.909035    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:45.083182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:45.083152    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:45.221463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:45.221385    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:45.433740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:45.433706    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:45.610320 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:45.610283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:45.789935 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:45.789857    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:45.938456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:45.938364    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:46.079980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.079888    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:46.110173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.110129    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sz5pz" event={"ID":"daddcc0b-25a9-4a95-af81-5bdf8e63a4b9","Type":"ContainerStarted","Data":"8cd053095cf1abc976f4256c30a1a5fe9a30f3da85a439cd7e7b15b0f54024ac"}
Nov 01 08:32:46.110464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.110445    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:46.121698 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.121645    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-sz5pz"
Nov 01 08:32:46.147935 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.147906    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8t4zj"]
Nov 01 08:32:46.153909 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.153885    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.230934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.230901    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:46.254124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.254075    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-sz5pz" podStartSLOduration=1.7368495419999999 podStartE2EDuration="17.254063484s" totalImagesPullingTime="15.517213942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:46.252947463 +0000 UTC m=+39.447342019" watchObservedRunningTime="2026-11-01 08:32:46.254063484 +0000 UTC m=+39.448458062"
Nov 01 08:32:46.259961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.259927    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-sys\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.259977    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-textfile\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.259995    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-metrics-client-ca\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.260077    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.260130    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-tls\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.260196    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-accelerators-collector-config\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.260275    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-wtmp\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260398 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.260326    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-root\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.260458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.260399    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvkz\" (UniqueName: \"kubernetes.io/projected/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-kube-api-access-nxvkz\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.291727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.291672    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 08:32:46.298942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.298875    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 08:32:46.299439 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.299415    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 08:32:46.299676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.299405    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 08:32:46.299676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.299485    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 08:32:46.299676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.299505    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 08:32:46.299676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.299556    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qwt9q\""
Nov 01 08:32:46.361274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361191    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-wtmp\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361247    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-root\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361280    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvkz\" (UniqueName: \"kubernetes.io/projected/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-kube-api-access-nxvkz\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361333    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-sys\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361344    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-root\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361367    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-textfile\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361391    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-metrics-client-ca\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361422    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361441    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-sys\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361492 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361452    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-tls\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361556    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-accelerators-collector-config\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361736    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-textfile\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.361947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.361922    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-wtmp\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.362167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.362139    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-metrics-client-ca\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.362376 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.362355    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-accelerators-collector-config\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.364074 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.364048    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-tls\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.364758 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.364733    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.373628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.373584    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:46.422128 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.422023    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvkz\" (UniqueName: \"kubernetes.io/projected/52d7169a-99e2-45ce-a22b-4454cd4d7fc1-kube-api-access-nxvkz\") pod \"node-exporter-8t4zj\" (UID: \"52d7169a-99e2-45ce-a22b-4454cd4d7fc1\") " pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.519868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.519828    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8t4zj"
Nov 01 08:32:46.536608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:46.536562    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:46.557246 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:46.557210    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d7169a_99e2_45ce_a22b_4454cd4d7fc1.slice/crio-ac5221b5dfa062441180f6c3542538b5377f499d9ef1acb38e1190383f4d10e0 WatchSource:0}: Error finding container ac5221b5dfa062441180f6c3542538b5377f499d9ef1acb38e1190383f4d10e0: Status 404 returned error can't find the container with id ac5221b5dfa062441180f6c3542538b5377f499d9ef1acb38e1190383f4d10e0
Nov 01 08:32:47.115801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:47.115759    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8t4zj" event={"ID":"52d7169a-99e2-45ce-a22b-4454cd4d7fc1","Type":"ContainerStarted","Data":"ac5221b5dfa062441180f6c3542538b5377f499d9ef1acb38e1190383f4d10e0"}
Nov 01 08:32:47.447509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:47.447422    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:47.616015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:47.615978    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:47.885542 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:47.885499    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:48.025736 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.025701    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:48.247472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.247435    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:48.649332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.649260    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-785787c595-mh66b"]
Nov 01 08:32:48.669244 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.669218    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.711523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.711496    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-785787c595-mh66b"]
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756629    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756653    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fjkp6r7cr2b5j\""
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756684    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756660    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756707    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756789    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 08:32:48.756874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.756854    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cnwcb\""
Nov 01 08:32:48.776560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776525    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87db2\" (UniqueName: \"kubernetes.io/projected/f475e2f8-e3ef-41f4-9671-03d69acdd652-kube-api-access-87db2\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776621    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776667    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f475e2f8-e3ef-41f4-9671-03d69acdd652-metrics-client-ca\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776741    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776782    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-tls\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776802    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f475e2f8-e3ef-41f4-9671-03d69acdd652-tmp\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776827    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-grpc-tls\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.776963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.776858    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877165    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87db2\" (UniqueName: \"kubernetes.io/projected/f475e2f8-e3ef-41f4-9671-03d69acdd652-kube-api-access-87db2\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877213    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877246    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877267    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f475e2f8-e3ef-41f4-9671-03d69acdd652-metrics-client-ca\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877301    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877337    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-tls\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877359    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f475e2f8-e3ef-41f4-9671-03d69acdd652-tmp\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877379    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-grpc-tls\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.877705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.877412    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.878562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.878405    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f475e2f8-e3ef-41f4-9671-03d69acdd652-tmp\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.882580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.882049    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.884545 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.882890    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.884545 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.883598    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f475e2f8-e3ef-41f4-9671-03d69acdd652-metrics-client-ca\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.888988 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.885291    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-grpc-tls\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.892514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.892486    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.895038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.895019    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.897691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.897669    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f475e2f8-e3ef-41f4-9671-03d69acdd652-secret-thanos-querier-tls\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:48.933932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:48.933904    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87db2\" (UniqueName: \"kubernetes.io/projected/f475e2f8-e3ef-41f4-9671-03d69acdd652-kube-api-access-87db2\") pod \"thanos-querier-785787c595-mh66b\" (UID: \"f475e2f8-e3ef-41f4-9671-03d69acdd652\") " pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:49.117421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:49.117384    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:49.125236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:49.125206    4314 generic.go:356] "Generic (PLEG): container finished" podID="52d7169a-99e2-45ce-a22b-4454cd4d7fc1" containerID="895aa66f1d0d832a28bd250568b59c2685151786013d804392b1769b77ece4df" exitCode=0
Nov 01 08:32:49.125364 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:49.125270    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8t4zj" event={"ID":"52d7169a-99e2-45ce-a22b-4454cd4d7fc1","Type":"ContainerDied","Data":"895aa66f1d0d832a28bd250568b59c2685151786013d804392b1769b77ece4df"}
Nov 01 08:32:49.556880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:49.556849    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-785787c595-mh66b"]
Nov 01 08:32:49.559948 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:49.559918    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf475e2f8_e3ef_41f4_9671_03d69acdd652.slice/crio-0a5e324d34b5c699e4d9ed1010ca04b27ce4efe1ac8cdc4dae089201cf6423f9 WatchSource:0}: Error finding container 0a5e324d34b5c699e4d9ed1010ca04b27ce4efe1ac8cdc4dae089201cf6423f9: Status 404 returned error can't find the container with id 0a5e324d34b5c699e4d9ed1010ca04b27ce4efe1ac8cdc4dae089201cf6423f9
Nov 01 08:32:50.092803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.092763    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:50.131671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.131640    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8t4zj" event={"ID":"52d7169a-99e2-45ce-a22b-4454cd4d7fc1","Type":"ContainerStarted","Data":"0bcc5f3fc572570228b64f5132a3e1b03fbc592f9300ce661b1bb72f2bf62e36"}
Nov 01 08:32:50.131828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.131700    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8t4zj" event={"ID":"52d7169a-99e2-45ce-a22b-4454cd4d7fc1","Type":"ContainerStarted","Data":"214491f9edbb37346950c2fe2d3ef9c8ae24e7ae3c6d5b8b389888eeb4f29e9b"}
Nov 01 08:32:50.133132 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.133096    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"0a5e324d34b5c699e4d9ed1010ca04b27ce4efe1ac8cdc4dae089201cf6423f9"}
Nov 01 08:32:50.243262 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.243211    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8t4zj" podStartSLOduration=2.023532054 podStartE2EDuration="4.243195763s" totalImagesPullingTime="2.123257239s" totalInitContainerRuntime="96.40647ms" isStatefulPod=true podCreationTimestamp="2026-11-01 08:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:50.242262825 +0000 UTC m=+43.436657379" watchObservedRunningTime="2026-11-01 08:32:50.243195763 +0000 UTC m=+43.437590317"
Nov 01 08:32:50.251247 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.251217    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:50.402313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.402237    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:50.542103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.542056    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:50.683914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.683839    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:50.855789 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.855757    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"]
Nov 01 08:32:50.858113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.858015    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:50.877208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.877183    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.915859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.915761    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"]
Nov 01 08:32:50.925682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.925392    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 08:32:50.926351 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.926285    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 08:32:50.960187 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.960128    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 08:32:50.960323 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.960220    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 08:32:50.996221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996188    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-secret-metrics-server-client-certs\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.996361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996230    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-audit-log\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.996361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996254    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.996361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996276    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzt6\" (UniqueName: \"kubernetes.io/projected/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-kube-api-access-jxzt6\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.996361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996302    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-metrics-server-audit-profiles\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.996361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996341    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-secret-metrics-server-tls\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:50.996617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:50.996363    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-client-ca-bundle\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.002126 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.002078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:51.017887 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.017815    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qvstp\""
Nov 01 08:32:51.018087 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.017907    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-did55gvve46sk\""
Nov 01 08:32:51.059217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.059166    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d4989f77b-l5wws"
Nov 01 08:32:51.097515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.097481    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-secret-metrics-server-tls\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.097590    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-client-ca-bundle\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.097978    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-secret-metrics-server-client-certs\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.098041    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-audit-log\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.098100    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098453 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.098199    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzt6\" (UniqueName: \"kubernetes.io/projected/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-kube-api-access-jxzt6\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098453 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.098245    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-metrics-server-audit-profiles\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.098707    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-audit-log\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.098940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.098905    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.099711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.099687    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-metrics-server-audit-profiles\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.101136 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.101119    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-secret-metrics-server-tls\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.103127 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.103090    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-client-ca-bundle\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.112564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.112525    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-secret-metrics-server-client-certs\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.146215 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.146162    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:51.161131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.161088    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzt6\" (UniqueName: \"kubernetes.io/projected/2fe94ec3-8a61-4a0e-be18-4a89125ffce8-kube-api-access-jxzt6\") pod \"metrics-server-646fcdc6b8-jpgtr\" (UID: \"2fe94ec3-8a61-4a0e-be18-4a89125ffce8\") " pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.208361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.208320    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:32:51.306598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.303238    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:51.504296 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.504216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:51.675077 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:51.675041    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"]
Nov 01 08:32:51.681285 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:51.681253    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe94ec3_8a61_4a0e_be18_4a89125ffce8.slice/crio-70f2644a42df9d01cb4798a0c286c0e02d3f7ce5bce003880842b2213fb729a9 WatchSource:0}: Error finding container 70f2644a42df9d01cb4798a0c286c0e02d3f7ce5bce003880842b2213fb729a9: Status 404 returned error can't find the container with id 70f2644a42df9d01cb4798a0c286c0e02d3f7ce5bce003880842b2213fb729a9
Nov 01 08:32:52.101165 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:52.101113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:52.148865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:52.148825    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr" event={"ID":"2fe94ec3-8a61-4a0e-be18-4a89125ffce8","Type":"ContainerStarted","Data":"70f2644a42df9d01cb4798a0c286c0e02d3f7ce5bce003880842b2213fb729a9"}
Nov 01 08:32:52.253007 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:52.252948    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:53.154599 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:53.154568    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"963d77f1c31ee9e803968a5e4957a20118c8c3dc038210ecbfd7bbf94b8beaef"}
Nov 01 08:32:53.294427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:53.294372    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:53.438085 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:53.438054    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:54.161713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:54.161670    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"5da35590f0ca68d80c57d895b31de7b9e39a6ee262ea1a2c6f9bf22caf045621"}
Nov 01 08:32:54.162079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:54.161757    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"040f6fd5a58d45a8c7266d2246998282f0ce30fb2de875a3d7d1063814043f8c"}
Nov 01 08:32:55.118959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.118930    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:55.166991 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.166963    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr" event={"ID":"2fe94ec3-8a61-4a0e-be18-4a89125ffce8","Type":"ContainerStarted","Data":"f2089c7957c84bb83f2fc513f7209d0a728190badf5b7267322403337bb2ca57"}
Nov 01 08:32:55.196504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.196478    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:32:55.213524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.213503    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.262970 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.262945    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:32:55.337926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.337903    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.337933    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.337952    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.337971    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vjq\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338064    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338096    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338113    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338129    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338234    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338270    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338291    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338312    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338327    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338412 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338367    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338412 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338387    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338420    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338445    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338527 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338474    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338527 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338514    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.338603 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.338553    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.349202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349170    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 08:32:55.349202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349170    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349205    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349214    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349220    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349258    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349271    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349278    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349288    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349325    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 08:32:55.349357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349325    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mk6s6\""
Nov 01 08:32:55.349657 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349395    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 08:32:55.349657 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.349439    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-51c6oksb1o85r\""
Nov 01 08:32:55.390338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.390288    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:55.393877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.393862    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 08:32:55.393988 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.393971    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 08:32:55.394149 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.394116    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr" podStartSLOduration=2.5155903459999998 podStartE2EDuration="5.394106209s" totalImagesPullingTime="2.878515863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:55.393988977 +0000 UTC m=+48.588383531" watchObservedRunningTime="2026-11-01 08:32:55.394106209 +0000 UTC m=+48.588500763"
Nov 01 08:32:55.394192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.394153    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 08:32:55.438931 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.438911    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.438939    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.438957    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vjq\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.438976    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.438994    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439018    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439363 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439344    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439347    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439415    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439446    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439473    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439507    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439541    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439567    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439590    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439615    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439641    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.439682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439669    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439712    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439740    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439744    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.439792    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440196 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.440072    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440196 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.440077    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440314 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.440293    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.440731 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.440699    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.442007 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.441829    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.443164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.443139    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.443261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.443205    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445062    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445259    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445272    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445307    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445364    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445806    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.445933 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.445898    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.446477 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.446459    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.446593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.446573    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.447019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.447004    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.495458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.495435    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vjq\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq\") pod \"prometheus-k8s-0\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:55.656431 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:55.656408    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:32:56.102566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:56.102523    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:32:56.105666 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:32:56.105634    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff924a8e_4e61_428d_9be2_14f0ead3cd8f.slice/crio-67a079b7a7de795fa5d5150afa5f7b18001711cd0bee02fc6e81ac03e9f469a4 WatchSource:0}: Error finding container 67a079b7a7de795fa5d5150afa5f7b18001711cd0bee02fc6e81ac03e9f469a4: Status 404 returned error can't find the container with id 67a079b7a7de795fa5d5150afa5f7b18001711cd0bee02fc6e81ac03e9f469a4
Nov 01 08:32:56.171457 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:56.171429    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"67a079b7a7de795fa5d5150afa5f7b18001711cd0bee02fc6e81ac03e9f469a4"}
Nov 01 08:32:56.174389 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:56.174355    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"a80e33302127caddd56d14a1083109d1b76e93d926d77a2b9227cc21aad6d340"}
Nov 01 08:32:56.174479 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:56.174405    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"7c73f30531334ecf0546ccdc4116d7f3ce4b80b9c43bc7447a1790fd2d7fce5b"}
Nov 01 08:32:56.174479 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:56.174432    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" event={"ID":"f475e2f8-e3ef-41f4-9671-03d69acdd652","Type":"ContainerStarted","Data":"3c960ce1b5e1d048669ea03ee51b0de90b2fe787c0d0a76ae862901548b0da05"}
Nov 01 08:32:57.177400 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:57.177369    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:58.185732 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:58.185703    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-785787c595-mh66b"
Nov 01 08:32:58.396309 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:58.394775    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-785787c595-mh66b" podStartSLOduration=4.885754171 podStartE2EDuration="10.39475679s" totalImagesPullingTime="5.509002619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:56.292584622 +0000 UTC m=+49.486979176" watchObservedRunningTime="2026-11-01 08:32:58.39475679 +0000 UTC m=+51.589151345"
Nov 01 08:32:59.184572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:59.184516    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc" exitCode=0
Nov 01 08:32:59.184758 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:59.184602    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc"}
Nov 01 08:32:59.204451 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:59.204432    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:32:59.347930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:32:59.347881    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:00.015930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:00.015895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:00.219756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:00.219683    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:02.086165 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:02.086127    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:02.232231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:02.232201    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:03.201707 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:03.201680    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8"}
Nov 01 08:33:04.208290 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.208258    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e"}
Nov 01 08:33:04.208739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.208320    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4"}
Nov 01 08:33:04.208739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.208350    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6"}
Nov 01 08:33:04.208739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.208377    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b"}
Nov 01 08:33:04.208739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.208400    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerStarted","Data":"da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492"}
Nov 01 08:33:04.315257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.315223    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:04.454571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.454521    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:04.470042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:04.469936    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.356423767 podStartE2EDuration="9.469923364s" totalImagesPullingTime="6.087563153s" totalInitContainerRuntime="25.936444ms" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:33:04.423038158 +0000 UTC m=+57.617432725" watchObservedRunningTime="2026-11-01 08:33:04.469923364 +0000 UTC m=+57.664317974"
Nov 01 08:33:05.656857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:05.656825    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:11.214386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:11.214335    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:33:11.214962 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:11.214421    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:33:11.321356 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:11.321320    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:33:12.699624 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:12.699589    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:12.840883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:12.840849    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:13.876317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:13.876279    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:14.017945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:14.017912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:16.024255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:16.024223    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:16.163901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:16.163868    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:21.468191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.468154    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5m778-tsfb9"]
Nov 01 08:33:21.468653 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.468430    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-5m778-tsfb9" podUID="a069a9b7-205f-4e32-8b62-0a9a1e67d59b" containerName="insights-gathering" containerID="cri-o://8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9" gracePeriod=30
Nov 01 08:33:21.478712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.478689    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:21.622776 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.622745    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:21.740462 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.740438    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:33:21.763145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763123    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\") pod \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") "
Nov 01 08:33:21.763261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763154    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\") pod \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") "
Nov 01 08:33:21.763261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763197    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\" (UniqueName: \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\") pod \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") "
Nov 01 08:33:21.763261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763226    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\") pod \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\" (UID: \"a069a9b7-205f-4e32-8b62-0a9a1e67d59b\") "
Nov 01 08:33:21.763474 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763447    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path" pod "a069a9b7-205f-4e32-8b62-0a9a1e67d59b" (UID: "a069a9b7-205f-4e32-8b62-0a9a1e67d59b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:33:21.763566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763473    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle" pod "a069a9b7-205f-4e32-8b62-0a9a1e67d59b" (UID: "a069a9b7-205f-4e32-8b62-0a9a1e67d59b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:21.763667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.763649    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle" pod "a069a9b7-205f-4e32-8b62-0a9a1e67d59b" (UID: "a069a9b7-205f-4e32-8b62-0a9a1e67d59b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:21.765311 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.765294    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht" pod "a069a9b7-205f-4e32-8b62-0a9a1e67d59b" (UID: "a069a9b7-205f-4e32-8b62-0a9a1e67d59b"). InnerVolumeSpecName "kube-api-access-wrcht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:33:21.863938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.863911    4314 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-archives-path\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:21.863938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.863935    4314 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-service-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:21.863938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.863944    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrcht\" (UniqueName: \"kubernetes.io/projected/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-kube-api-access-wrcht\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:21.864158 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:21.863955    4314 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a069a9b7-205f-4e32-8b62-0a9a1e67d59b-trusted-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:22.270845 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.270814    4314 generic.go:356] "Generic (PLEG): container finished" podID="a069a9b7-205f-4e32-8b62-0a9a1e67d59b" containerID="8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9" exitCode=2
Nov 01 08:33:22.271053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.270855    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m778-tsfb9"
Nov 01 08:33:22.271053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.270884    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m778-tsfb9" event={"ID":"a069a9b7-205f-4e32-8b62-0a9a1e67d59b","Type":"ContainerDied","Data":"8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9"}
Nov 01 08:33:22.271053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.270978    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m778-tsfb9" event={"ID":"a069a9b7-205f-4e32-8b62-0a9a1e67d59b","Type":"ContainerDied","Data":"03f9d9758c56585ff23765b80a9b3b4f74b5f238b2d3b7efae45e6c2e3e859a9"}
Nov 01 08:33:22.271053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.271012    4314 scope.go:122] "RemoveContainer" containerID="8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9"
Nov 01 08:33:22.286103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.286088    4314 scope.go:122] "RemoveContainer" containerID="8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9"
Nov 01 08:33:22.286354 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:22.286330    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9\": container with ID starting with 8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9 not found: ID does not exist" containerID="8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9"
Nov 01 08:33:22.286456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.286360    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9"} err="failed to get container status \"8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9\": rpc error: code = NotFound desc = could not find container \"8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9\": container with ID starting with 8e29fc0e6a235d6f05897f4db8486bd5e015e717ce1043e9bc9a1fa2eac921b9 not found: ID does not exist"
Nov 01 08:33:22.422428 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.422399    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5m778-tsfb9"]
Nov 01 08:33:22.468700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.468673    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-5m778-tsfb9"]
Nov 01 08:33:22.925848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:22.925819    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a069a9b7-205f-4e32-8b62-0a9a1e67d59b" path="/var/lib/kubelet/pods/a069a9b7-205f-4e32-8b62-0a9a1e67d59b/volumes"
Nov 01 08:33:24.212670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:24.212639    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:24.359741 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:24.359709    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:25.296095 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:25.296063    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:25.340591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:25.340567    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:25.672159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:25.672091    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:28.879460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:28.879413    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:29.018419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:29.018383    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:31.214224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:31.214195    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:33:31.217899 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:31.217877    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-646fcdc6b8-jpgtr"
Nov 01 08:33:39.458442 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:39.458404    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:39.657809 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:39.657770    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:41.322243 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:41.322198    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:33:47.233033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.232998    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:33:47.233597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.233268    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="prometheus" containerID="cri-o://f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8" gracePeriod=600
Nov 01 08:33:47.233597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.233313    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e" gracePeriod=600
Nov 01 08:33:47.233597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.233303    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="thanos-sidecar" containerID="cri-o://f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b" gracePeriod=600
Nov 01 08:33:47.233597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.233350    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-web" containerID="cri-o://f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6" gracePeriod=600
Nov 01 08:33:47.233597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.233350    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="config-reloader" containerID="cri-o://da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492" gracePeriod=600
Nov 01 08:33:47.233597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.233285    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy" containerID="cri-o://ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4" gracePeriod=600
Nov 01 08:33:47.361642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361609    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e" exitCode=0
Nov 01 08:33:47.361642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361639    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4" exitCode=0
Nov 01 08:33:47.361642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361646    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6" exitCode=0
Nov 01 08:33:47.361838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361651    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b" exitCode=0
Nov 01 08:33:47.361838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361656    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492" exitCode=0
Nov 01 08:33:47.361838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361684    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e"}
Nov 01 08:33:47.361838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361764    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4"}
Nov 01 08:33:47.361838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361800    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6"}
Nov 01 08:33:47.361838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361836    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b"}
Nov 01 08:33:47.362008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.361872    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492"}
Nov 01 08:33:47.584211 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.584186    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:47.659448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659411    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659456    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659487    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659513    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659567    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659620    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659661    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659687    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659713    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659750    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.659787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659782    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659819    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659847    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659900    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659940    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.659969    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.660027    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.660068    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.660100    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.660148    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1\") pod \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\" (UID: \"ff924a8e-4e61-428d-9be2-14f0ead3cd8f\") "
Nov 01 08:33:47.660957 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.660702    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.661120 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.661067    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.662062    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.662494    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.662742    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.662756    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.662816    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.662920    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "kube-api-access-x7vjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.663064    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.663026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.663213    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:47.663879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.663263    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.663879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.663345    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.663879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.663410    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.663879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.663831    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.664733 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.664709    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.664834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.664794    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:33:47.665116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.665094    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.665258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.665239    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.666091 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.666066    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:33:47.673331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.673308    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config" pod "ff924a8e-4e61-428d-9be2-14f0ead3cd8f" (UID: "ff924a8e-4e61-428d-9be2-14f0ead3cd8f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761376    4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761403    4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761414    4314 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761424    4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761434    4314 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761443    4314 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-kube-rbac-proxy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761452    4314 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-web-config\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761460    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vjq\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-kube-api-access-x7vjq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761469    4314 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-grpc-tls\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761477    4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-db\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761488    4314 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-metrics-client-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761498    4314 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761507    4314 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-metrics-client-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761515    4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761524    4314 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761543    4314 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-config-out\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761552    4314 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761560    4314 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761568    4314 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-tls-assets\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:47.761787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:47.761576    4314 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff924a8e-4e61-428d-9be2-14f0ead3cd8f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:48.367984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.367951    4314 generic.go:356] "Generic (PLEG): container finished" podID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerID="f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8" exitCode=0
Nov 01 08:33:48.368386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.368009    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8"}
Nov 01 08:33:48.368386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.368034    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.368386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.368066    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff924a8e-4e61-428d-9be2-14f0ead3cd8f","Type":"ContainerDied","Data":"67a079b7a7de795fa5d5150afa5f7b18001711cd0bee02fc6e81ac03e9f469a4"}
Nov 01 08:33:48.368386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.368097    4314 scope.go:122] "RemoveContainer" containerID="a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e"
Nov 01 08:33:48.375660 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.375587    4314 scope.go:122] "RemoveContainer" containerID="ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4"
Nov 01 08:33:48.382169 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.382150    4314 scope.go:122] "RemoveContainer" containerID="f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6"
Nov 01 08:33:48.388432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.388409    4314 scope.go:122] "RemoveContainer" containerID="f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b"
Nov 01 08:33:48.394710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.394691    4314 scope.go:122] "RemoveContainer" containerID="da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492"
Nov 01 08:33:48.400961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.400941    4314 scope.go:122] "RemoveContainer" containerID="f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8"
Nov 01 08:33:48.407685 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.407666    4314 scope.go:122] "RemoveContainer" containerID="037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc"
Nov 01 08:33:48.413967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.413950    4314 scope.go:122] "RemoveContainer" containerID="a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e"
Nov 01 08:33:48.414234 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.414218    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e\": container with ID starting with a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e not found: ID does not exist" containerID="a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e"
Nov 01 08:33:48.414277 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414242    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e"} err="failed to get container status \"a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e\": rpc error: code = NotFound desc = could not find container \"a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e\": container with ID starting with a279b78009ae0dcd6c1d28e44d29ef72a6703fafd83766accdceea94a649b00e not found: ID does not exist"
Nov 01 08:33:48.414277 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414262    4314 scope.go:122] "RemoveContainer" containerID="ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4"
Nov 01 08:33:48.414479 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.414461    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4\": container with ID starting with ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4 not found: ID does not exist" containerID="ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4"
Nov 01 08:33:48.414519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414484    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4"} err="failed to get container status \"ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4\": rpc error: code = NotFound desc = could not find container \"ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4\": container with ID starting with ef32ecdd53604a3bf9a5f195994631a2b4602d9eae8a6acd8ec2420e23bb65e4 not found: ID does not exist"
Nov 01 08:33:48.414519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414499    4314 scope.go:122] "RemoveContainer" containerID="f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6"
Nov 01 08:33:48.414722 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.414706    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6\": container with ID starting with f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6 not found: ID does not exist" containerID="f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6"
Nov 01 08:33:48.414801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414726    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6"} err="failed to get container status \"f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6\": rpc error: code = NotFound desc = could not find container \"f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6\": container with ID starting with f7095965d85cc61f1a0c4001bdc981e67ecc568e3bbc418e44565c1e106be4c6 not found: ID does not exist"
Nov 01 08:33:48.414801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414740    4314 scope.go:122] "RemoveContainer" containerID="f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b"
Nov 01 08:33:48.414980 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.414962    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b\": container with ID starting with f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b not found: ID does not exist" containerID="f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b"
Nov 01 08:33:48.415019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.414986    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b"} err="failed to get container status \"f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b\": rpc error: code = NotFound desc = could not find container \"f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b\": container with ID starting with f75424831240e2d10a67e4dc22a1c1b45cc45dfd2e7faf1c3af471d59e72ef7b not found: ID does not exist"
Nov 01 08:33:48.415019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.415002    4314 scope.go:122] "RemoveContainer" containerID="da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492"
Nov 01 08:33:48.415209 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.415190    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492\": container with ID starting with da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492 not found: ID does not exist" containerID="da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492"
Nov 01 08:33:48.415250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.415218    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492"} err="failed to get container status \"da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492\": rpc error: code = NotFound desc = could not find container \"da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492\": container with ID starting with da067690cba30d8320a22923b08174c6d9940556ac4d6a5859401d89ab2f1492 not found: ID does not exist"
Nov 01 08:33:48.415250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.415237    4314 scope.go:122] "RemoveContainer" containerID="f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8"
Nov 01 08:33:48.415467 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.415447    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8\": container with ID starting with f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8 not found: ID does not exist" containerID="f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8"
Nov 01 08:33:48.415545 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.415475    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8"} err="failed to get container status \"f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8\": rpc error: code = NotFound desc = could not find container \"f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8\": container with ID starting with f21585d0d2838b4666c997f1b8beb06790f5f9270fd1dbb9409dffd5f40ffde8 not found: ID does not exist"
Nov 01 08:33:48.415545 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.415494    4314 scope.go:122] "RemoveContainer" containerID="037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc"
Nov 01 08:33:48.415752 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:33:48.415736    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc\": container with ID starting with 037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc not found: ID does not exist" containerID="037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc"
Nov 01 08:33:48.415790 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.415756    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc"} err="failed to get container status \"037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc\": rpc error: code = NotFound desc = could not find container \"037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc\": container with ID starting with 037a276b5c5e843b0124c83c3de85965d78acb7d82e003b4a4faaca24959a9fc not found: ID does not exist"
Nov 01 08:33:48.569715 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.569630    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:33:48.615964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.615935    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:33:48.616695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.616674    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:33:48.616963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.616948    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy"
Nov 01 08:33:48.617032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.616966    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy"
Nov 01 08:33:48.617032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.616987    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="config-reloader"
Nov 01 08:33:48.617032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.616995    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="config-reloader"
Nov 01 08:33:48.617032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617011    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="init-config-reloader"
Nov 01 08:33:48.617032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617020    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="init-config-reloader"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617039    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-thanos"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617047    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-thanos"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617061    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="prometheus"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617069    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="prometheus"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617079    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="thanos-sidecar"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617086    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="thanos-sidecar"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617098    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-web"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617105    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-web"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617119    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a069a9b7-205f-4e32-8b62-0a9a1e67d59b" containerName="insights-gathering"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617126    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a069a9b7-205f-4e32-8b62-0a9a1e67d59b" containerName="insights-gathering"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617193    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="config-reloader"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617210    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-thanos"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617220    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="prometheus"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617228    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="thanos-sidecar"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617236    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy-web"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617246    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a069a9b7-205f-4e32-8b62-0a9a1e67d59b" containerName="insights-gathering"
Nov 01 08:33:48.617265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.617256    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" containerName="kube-rbac-proxy"
Nov 01 08:33:48.620553 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.620515    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668262 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668226    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668262 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668262    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668295    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668337    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668358    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668455    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33087e46-8c7e-4d5a-a7d3-888da30bb31d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668472    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-config\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668505    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668524    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668696    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668721    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbmq\" (UniqueName: \"kubernetes.io/projected/33087e46-8c7e-4d5a-a7d3-888da30bb31d-kube-api-access-rxbmq\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668764    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668915 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668816    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668915 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668885    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.668915 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668912    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.669037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668977    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-web-config\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.669037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.668999    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.669133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669065    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33087e46-8c7e-4d5a-a7d3-888da30bb31d-config-out\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.669133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669097    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.669614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669593    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 08:33:48.669708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669602    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 08:33:48.669708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669640    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 08:33:48.669708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669662    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 08:33:48.669708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669665    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 08:33:48.669860 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669770    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 08:33:48.669860 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669838    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 08:33:48.669955 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669904    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 08:33:48.669955 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669918    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 08:33:48.669955 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669940    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 08:33:48.670053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.669954    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-51c6oksb1o85r\""
Nov 01 08:33:48.670053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.670042    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 08:33:48.670053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.670048    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 08:33:48.670180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.670169    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mk6s6\""
Nov 01 08:33:48.716584 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.716560    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 08:33:48.716697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.716630    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 08:33:48.770204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770174    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770205    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770223    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770255    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33087e46-8c7e-4d5a-a7d3-888da30bb31d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770283    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-config\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770404    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770447    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770480    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770549    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770578    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbmq\" (UniqueName: \"kubernetes.io/projected/33087e46-8c7e-4d5a-a7d3-888da30bb31d-kube-api-access-rxbmq\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770622    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770646    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770674    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770697    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.770891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.770737    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-web-config\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.771189    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.771228    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33087e46-8c7e-4d5a-a7d3-888da30bb31d-config-out\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.771256    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.771297    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.771323    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.772261    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.772972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.772830    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.773681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.773398    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.773681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.773419    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.774021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.773996    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-config\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.774710    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-web-config\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.774842    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775042    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775332    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775573    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775688    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775818    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775837    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.775937    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33087e46-8c7e-4d5a-a7d3-888da30bb31d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.776233    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.776328    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33087e46-8c7e-4d5a-a7d3-888da30bb31d-config-out\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776794 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.776728    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776794 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.776759    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33087e46-8c7e-4d5a-a7d3-888da30bb31d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.776933 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.776917    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33087e46-8c7e-4d5a-a7d3-888da30bb31d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.827411 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.827359    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbmq\" (UniqueName: \"kubernetes.io/projected/33087e46-8c7e-4d5a-a7d3-888da30bb31d-kube-api-access-rxbmq\") pod \"prometheus-k8s-0\" (UID: \"33087e46-8c7e-4d5a-a7d3-888da30bb31d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:48.842674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.842653    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:33:48.925885 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.925849    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff924a8e-4e61-428d-9be2-14f0ead3cd8f" path="/var/lib/kubelet/pods/ff924a8e-4e61-428d-9be2-14f0ead3cd8f/volumes"
Nov 01 08:33:48.937212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:48.937188    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:49.353282 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:49.353089    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 08:33:49.356094 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:33:49.356066    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33087e46_8c7e_4d5a_a7d3_888da30bb31d.slice/crio-249d3cda0b7793c44bd230ed3c8d8f4135341b5286351fad13b020dd8de25cd0 WatchSource:0}: Error finding container 249d3cda0b7793c44bd230ed3c8d8f4135341b5286351fad13b020dd8de25cd0: Status 404 returned error can't find the container with id 249d3cda0b7793c44bd230ed3c8d8f4135341b5286351fad13b020dd8de25cd0
Nov 01 08:33:49.373721 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:49.373693    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"249d3cda0b7793c44bd230ed3c8d8f4135341b5286351fad13b020dd8de25cd0"}
Nov 01 08:33:50.226582 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:50.226548    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:50.376842 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:50.376810    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:50.379951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:50.379926    4314 generic.go:356] "Generic (PLEG): container finished" podID="33087e46-8c7e-4d5a-a7d3-888da30bb31d" containerID="1c1a8b424fe9a7d0a2663abaade971f3cc26a2346a05c0553ffb2342a0d618e2" exitCode=0
Nov 01 08:33:50.380096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:50.380001    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerDied","Data":"1c1a8b424fe9a7d0a2663abaade971f3cc26a2346a05c0553ffb2342a0d618e2"}
Nov 01 08:33:50.626673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:50.626577    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:50.781605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:50.781572    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:33:51.386324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.386283    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"0661e841aacfd75cdfbbb2ff6ab2dd3f762ffe29650cef6cf9c5daa3403f14e4"}
Nov 01 08:33:51.386723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.386369    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"3443dcef2544a2fe08f5e84c4d7016fd74d6448956209e74ad1e03feea972069"}
Nov 01 08:33:51.386723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.386413    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"083c43bc5e45f9387e79279914362b8dfbcc5766c79ff626f2b78386b60ac384"}
Nov 01 08:33:51.386723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.386448    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"b041df50fd8875459913da41a6681e0f2b03ade4a62ccb8ecb2744be0b0646a3"}
Nov 01 08:33:51.386723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.386485    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"21f0fff3a47f20e4e569e5392b727fe2e6ea9fac181f1c70d80a38a4c1d94817"}
Nov 01 08:33:51.386723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.386519    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33087e46-8c7e-4d5a-a7d3-888da30bb31d","Type":"ContainerStarted","Data":"62e1864ceaba69f4c670ea43b5fea2912e8a388f7a4fd4af6631bbc68f3ca231"}
Nov 01 08:33:51.510922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:51.510858    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.494668875 podStartE2EDuration="3.510840827s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.171952ms" isStatefulPod=false podCreationTimestamp="2026-11-01 08:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:33:51.509463129 +0000 UTC m=+104.703857666" watchObservedRunningTime="2026-11-01 08:33:51.510840827 +0000 UTC m=+104.705235383"
Nov 01 08:33:53.938340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:53.938294    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:33:58.758616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:33:58.758578    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:34:00.343980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:34:00.343942    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:34:11.320781 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:34:11.320744    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:34:41.321320 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:34:41.321237    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:34:48.938017 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:34:48.937986    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:34:48.953223 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:34:48.953198    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:34:49.597425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:34:49.597401    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 08:34:57.111334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:34:57.111299    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:34:57.250071 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:34:57.250042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:00.673209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:00.673175    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:09.942812 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:09.942762    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:10.088184 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:10.088133    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:11.320734 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:35:11.320701    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:35:13.882511 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:13.882478    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:14.024410 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:14.024376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:24.135158 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:24.135126    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:24.287684 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:24.287638    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:25.406439 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:25.406386    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:25.553040 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:25.553007    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:33.594248 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:35:33.594210    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:35:41.321456 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:35:41.321419    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:36:01.009674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:36:01.009584    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:36:11.320371 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:36:11.320337    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:36:41.320808 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:36:41.320765    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:37:01.347694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:01.347662    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:02.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:02.504958    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:08.925938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:08.925897    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:09.094468 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:09.094431    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:10.349315 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:10.349278    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:10.493071 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:10.493042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:11.321134 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:37:11.321102    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:37:11.638133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:11.638036    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:16.856696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:16.856660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:17.003031 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:17.003000    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:18.222933 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:18.222895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:18.495953 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:18.495923    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:18.637723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:18.637689    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:19.302258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.302227    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"]
Nov 01 08:37:19.305569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.305554    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.332380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.332358    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"]
Nov 01 08:37:19.335595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.335581    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.354049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.354028    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 08:37:19.377485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.377466    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mlfjl\""
Nov 01 08:37:19.377485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.377481    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 08:37:19.378393 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.378376    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"]
Nov 01 08:37:19.398947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.398925    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.399018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.398954    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.399018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.398978    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdmr\" (UniqueName: \"kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.401213 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.401192    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"]
Nov 01 08:37:19.404860 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.404846    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"]
Nov 01 08:37:19.404917 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.404874    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"]
Nov 01 08:37:19.404973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.404959    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.408104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.408083    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.429560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.429522    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"]
Nov 01 08:37:19.455458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.455442    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-fq9vb\""
Nov 01 08:37:19.455581 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.455568    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pc49r\""
Nov 01 08:37:19.457578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.457566    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8mn7l\""
Nov 01 08:37:19.459105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.459082    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"]
Nov 01 08:37:19.499764 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499742    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.499897 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499768    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.499897 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499788    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrgp\" (UniqueName: \"kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.499970 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499926    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prltn\" (UniqueName: \"kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.499970 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499949    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.500042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499968    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.500042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.499983    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.500042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500008    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdmr\" (UniqueName: \"kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.500238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500219    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.500307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500269    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4n78\" (UniqueName: \"kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.500357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500322    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.500357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500342    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.500432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500359    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.500432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.500374    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.557188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.557129    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdmr\" (UniqueName: \"kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.601508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601482    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.601642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601523    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4n78\" (UniqueName: \"kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.601642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601576    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.601642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601599    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.601813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601713    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.601813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601757    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.601813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601786    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrgp\" (UniqueName: \"kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.601964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601844    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.601964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601886    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prltn\" (UniqueName: \"kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.601964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601913    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.601964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.601936    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.602157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.602034    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.602157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.602095    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.602157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.602148    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.602295 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.602278    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.620979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.620962    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:19.659120 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.659091    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrgp\" (UniqueName: \"kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.659284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.659168    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prltn\" (UniqueName: \"kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.659284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.659241    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4n78\" (UniqueName: \"kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.721191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.721165    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:19.727237 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.727220    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:19.950614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.950548    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:19.983132 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.983111    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"]
Nov 01 08:37:19.985487 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:37:19.985455    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ccaed6_96d7_4be0_accf_b00784a7d6e2.slice/crio-ea7ffd3f5d754be6460061ae14bf6f3bc8189974247074608cafd4109e59f3f8 WatchSource:0}: Error finding container ea7ffd3f5d754be6460061ae14bf6f3bc8189974247074608cafd4109e59f3f8: Status 404 returned error can't find the container with id ea7ffd3f5d754be6460061ae14bf6f3bc8189974247074608cafd4109e59f3f8
Nov 01 08:37:19.988064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:19.988046    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:37:20.085787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:20.085759    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"]
Nov 01 08:37:20.087677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:20.087645    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" event={"ID":"95ccaed6-96d7-4be0-accf-b00784a7d6e2","Type":"ContainerStarted","Data":"ea7ffd3f5d754be6460061ae14bf6f3bc8189974247074608cafd4109e59f3f8"}
Nov 01 08:37:20.088389 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:37:20.088351    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc612aa0_e69c_4ad4_8b03_af18a7841d5e.slice/crio-c46e49312a7f5d981d4c6e0493fc069a7a76ffa66cf7e126a33ca1cdfdf14231 WatchSource:0}: Error finding container c46e49312a7f5d981d4c6e0493fc069a7a76ffa66cf7e126a33ca1cdfdf14231: Status 404 returned error can't find the container with id c46e49312a7f5d981d4c6e0493fc069a7a76ffa66cf7e126a33ca1cdfdf14231
Nov 01 08:37:20.089418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:20.089397    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"]
Nov 01 08:37:20.093513 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:37:20.093493    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045647ae_d63f_4d63_9188_b624fa59f605.slice/crio-d6af6860b487dc902e54b39f399408cde1d0b0bf6da6faca9b07e78e621d92c0 WatchSource:0}: Error finding container d6af6860b487dc902e54b39f399408cde1d0b0bf6da6faca9b07e78e621d92c0: Status 404 returned error can't find the container with id d6af6860b487dc902e54b39f399408cde1d0b0bf6da6faca9b07e78e621d92c0
Nov 01 08:37:20.311395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:20.311315    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"]
Nov 01 08:37:20.313974 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:37:20.313948    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bac9f9_cb0e_4388_a9f3_f68f926308dc.slice/crio-5c46e787c94fffad6300813162958e1924fc284bced21adfb9c810b5b661d5ce WatchSource:0}: Error finding container 5c46e787c94fffad6300813162958e1924fc284bced21adfb9c810b5b661d5ce: Status 404 returned error can't find the container with id 5c46e787c94fffad6300813162958e1924fc284bced21adfb9c810b5b661d5ce
Nov 01 08:37:21.094154 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:21.094117    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn" event={"ID":"cc612aa0-e69c-4ad4-8b03-af18a7841d5e","Type":"ContainerStarted","Data":"c46e49312a7f5d981d4c6e0493fc069a7a76ffa66cf7e126a33ca1cdfdf14231"}
Nov 01 08:37:21.095598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:21.095568    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9" event={"ID":"045647ae-d63f-4d63-9188-b624fa59f605","Type":"ContainerStarted","Data":"d6af6860b487dc902e54b39f399408cde1d0b0bf6da6faca9b07e78e621d92c0"}
Nov 01 08:37:21.097045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:21.097009    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l" event={"ID":"e1bac9f9-cb0e-4388-a9f3-f68f926308dc","Type":"ContainerStarted","Data":"5c46e787c94fffad6300813162958e1924fc284bced21adfb9c810b5b661d5ce"}
Nov 01 08:37:22.339111 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:22.339072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:30.142928 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.142893    4314 generic.go:356] "Generic (PLEG): container finished" podID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerID="6c5e28d585f674672462c55004d49d85ca21caff9a23fe2c4972ad6bb4666deb" exitCode=0
Nov 01 08:37:30.143337 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.142965    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" event={"ID":"95ccaed6-96d7-4be0-accf-b00784a7d6e2","Type":"ContainerDied","Data":"6c5e28d585f674672462c55004d49d85ca21caff9a23fe2c4972ad6bb4666deb"}
Nov 01 08:37:30.144523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.144500    4314 generic.go:356] "Generic (PLEG): container finished" podID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerID="0794447d78619e63298363d8b91bd7bcc5d1d3fe902fc1b65b8b5799eb176273" exitCode=0
Nov 01 08:37:30.144636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.144588    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l" event={"ID":"e1bac9f9-cb0e-4388-a9f3-f68f926308dc","Type":"ContainerDied","Data":"0794447d78619e63298363d8b91bd7bcc5d1d3fe902fc1b65b8b5799eb176273"}
Nov 01 08:37:30.146196 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.146163    4314 generic.go:356] "Generic (PLEG): container finished" podID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerID="590db06bc7420c10f433b7504f89911e0ad69b7d1d0d569d54d5d70a19927e51" exitCode=0
Nov 01 08:37:30.146261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.146186    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn" event={"ID":"cc612aa0-e69c-4ad4-8b03-af18a7841d5e","Type":"ContainerDied","Data":"590db06bc7420c10f433b7504f89911e0ad69b7d1d0d569d54d5d70a19927e51"}
Nov 01 08:37:30.147711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.147686    4314 generic.go:356] "Generic (PLEG): container finished" podID="045647ae-d63f-4d63-9188-b624fa59f605" containerID="4664a6f5bf2b24e79e4cd5705d5744dcce474d229021a418e593d92be02cc0b1" exitCode=0
Nov 01 08:37:30.147827 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:30.147709    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9" event={"ID":"045647ae-d63f-4d63-9188-b624fa59f605","Type":"ContainerDied","Data":"4664a6f5bf2b24e79e4cd5705d5744dcce474d229021a418e593d92be02cc0b1"}
Nov 01 08:37:33.160142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:33.160110    4314 generic.go:356] "Generic (PLEG): container finished" podID="045647ae-d63f-4d63-9188-b624fa59f605" containerID="08f49d240fe14990f9be5c2512d079258f5a935a1db946727b97a2c4638a46eb" exitCode=0
Nov 01 08:37:33.160579 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:33.160199    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9" event={"ID":"045647ae-d63f-4d63-9188-b624fa59f605","Type":"ContainerDied","Data":"08f49d240fe14990f9be5c2512d079258f5a935a1db946727b97a2c4638a46eb"}
Nov 01 08:37:33.161888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:33.161866    4314 generic.go:356] "Generic (PLEG): container finished" podID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerID="0319c8c13b93315d1de77a612f073867c1123023593335e942127a77da36f859" exitCode=0
Nov 01 08:37:33.162004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:33.161935    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" event={"ID":"95ccaed6-96d7-4be0-accf-b00784a7d6e2","Type":"ContainerDied","Data":"0319c8c13b93315d1de77a612f073867c1123023593335e942127a77da36f859"}
Nov 01 08:37:33.163744 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:33.163684    4314 generic.go:356] "Generic (PLEG): container finished" podID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerID="84ac0a2577f31faadd8b626e8f00e8c7cb8ef9bc2aece2e3342f6d34563938e1" exitCode=0
Nov 01 08:37:33.163744 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:33.163725    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l" event={"ID":"e1bac9f9-cb0e-4388-a9f3-f68f926308dc","Type":"ContainerDied","Data":"84ac0a2577f31faadd8b626e8f00e8c7cb8ef9bc2aece2e3342f6d34563938e1"}
Nov 01 08:37:34.171748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:34.170816    4314 generic.go:356] "Generic (PLEG): container finished" podID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerID="9405a276697e5c962e1acff013e9021fa6db4c99d38d9acf95ab46aa740fdd91" exitCode=0
Nov 01 08:37:34.171748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:34.170914    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn" event={"ID":"cc612aa0-e69c-4ad4-8b03-af18a7841d5e","Type":"ContainerDied","Data":"9405a276697e5c962e1acff013e9021fa6db4c99d38d9acf95ab46aa740fdd91"}
Nov 01 08:37:41.337225 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:37:41.337168    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:37:43.228951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:43.228872    4314 generic.go:356] "Generic (PLEG): container finished" podID="045647ae-d63f-4d63-9188-b624fa59f605" containerID="33d44b5594249bca1ddc6a4e2bf430623bc4d8e61f8e4ee39d19edc3148fc8ee" exitCode=0
Nov 01 08:37:43.229316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:43.228953    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9" event={"ID":"045647ae-d63f-4d63-9188-b624fa59f605","Type":"ContainerDied","Data":"33d44b5594249bca1ddc6a4e2bf430623bc4d8e61f8e4ee39d19edc3148fc8ee"}
Nov 01 08:37:43.231316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:43.231283    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" event={"ID":"95ccaed6-96d7-4be0-accf-b00784a7d6e2","Type":"ContainerStarted","Data":"cd0fe0511d865e40994ff27422c112f43370477b1238be9a1526be9e6efba5fd"}
Nov 01 08:37:43.233918 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:43.233894    4314 generic.go:356] "Generic (PLEG): container finished" podID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerID="8ac67e8a12a1827284520ebe600a8b67fdebbfd033fd0b2d552ee716bca6da20" exitCode=0
Nov 01 08:37:43.234016 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:43.233952    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l" event={"ID":"e1bac9f9-cb0e-4388-a9f3-f68f926308dc","Type":"ContainerDied","Data":"8ac67e8a12a1827284520ebe600a8b67fdebbfd033fd0b2d552ee716bca6da20"}
Nov 01 08:37:43.559998 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:43.559942    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" podStartSLOduration=3.11452588 podStartE2EDuration="24.559929521s" totalImagesPullingTime="21.445403641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:43.559744839 +0000 UTC m=+336.754139411" watchObservedRunningTime="2026-11-01 08:37:43.559929521 +0000 UTC m=+336.754324075"
Nov 01 08:37:44.239324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.239296    4314 generic.go:356] "Generic (PLEG): container finished" podID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerID="cd0fe0511d865e40994ff27422c112f43370477b1238be9a1526be9e6efba5fd" exitCode=0
Nov 01 08:37:44.239763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.239391    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" event={"ID":"95ccaed6-96d7-4be0-accf-b00784a7d6e2","Type":"ContainerDied","Data":"cd0fe0511d865e40994ff27422c112f43370477b1238be9a1526be9e6efba5fd"}
Nov 01 08:37:44.241250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.241225    4314 generic.go:356] "Generic (PLEG): container finished" podID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerID="55384132fd46efc19b828bab0fbe580b960d1beb0aba74e30a42f92c78635b0a" exitCode=0
Nov 01 08:37:44.241343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.241295    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn" event={"ID":"cc612aa0-e69c-4ad4-8b03-af18a7841d5e","Type":"ContainerDied","Data":"55384132fd46efc19b828bab0fbe580b960d1beb0aba74e30a42f92c78635b0a"}
Nov 01 08:37:44.370155 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.370129    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:44.380186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.380159    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:44.436752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.436728    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle\") pod \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") "
Nov 01 08:37:44.436857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.436763    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle\") pod \"045647ae-d63f-4d63-9188-b624fa59f605\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") "
Nov 01 08:37:44.436857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.436790    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78\" (UniqueName: \"kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78\") pod \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") "
Nov 01 08:37:44.436857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.436813    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util\") pod \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\" (UID: \"e1bac9f9-cb0e-4388-a9f3-f68f926308dc\") "
Nov 01 08:37:44.436857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.436846    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util\") pod \"045647ae-d63f-4d63-9188-b624fa59f605\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") "
Nov 01 08:37:44.437039 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.436873    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn\" (UniqueName: \"kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn\") pod \"045647ae-d63f-4d63-9188-b624fa59f605\" (UID: \"045647ae-d63f-4d63-9188-b624fa59f605\") "
Nov 01 08:37:44.437445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.437412    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle" pod "045647ae-d63f-4d63-9188-b624fa59f605" (UID: "045647ae-d63f-4d63-9188-b624fa59f605"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:44.437598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.437581    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle" pod "e1bac9f9-cb0e-4388-a9f3-f68f926308dc" (UID: "e1bac9f9-cb0e-4388-a9f3-f68f926308dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:44.439042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.439019    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn" pod "045647ae-d63f-4d63-9188-b624fa59f605" (UID: "045647ae-d63f-4d63-9188-b624fa59f605"). InnerVolumeSpecName "kube-api-access-prltn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:44.439144 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.439130    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78" pod "e1bac9f9-cb0e-4388-a9f3-f68f926308dc" (UID: "e1bac9f9-cb0e-4388-a9f3-f68f926308dc"). InnerVolumeSpecName "kube-api-access-l4n78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:44.442566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.442525    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util" pod "045647ae-d63f-4d63-9188-b624fa59f605" (UID: "045647ae-d63f-4d63-9188-b624fa59f605"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:44.444515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.444444    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util" pod "e1bac9f9-cb0e-4388-a9f3-f68f926308dc" (UID: "e1bac9f9-cb0e-4388-a9f3-f68f926308dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:44.537992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.537937    4314 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:44.537992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.537961    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4n78\" (UniqueName: \"kubernetes.io/projected/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-kube-api-access-l4n78\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:44.537992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.537975    4314 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-util\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:44.537992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.537987    4314 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045647ae-d63f-4d63-9188-b624fa59f605-util\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:44.538178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.537999    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prltn\" (UniqueName: \"kubernetes.io/projected/045647ae-d63f-4d63-9188-b624fa59f605-kube-api-access-prltn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:44.538178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:44.538013    4314 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1bac9f9-cb0e-4388-a9f3-f68f926308dc-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:45.246857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.246827    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l"
Nov 01 08:37:45.246857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.246840    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adnn5l" event={"ID":"e1bac9f9-cb0e-4388-a9f3-f68f926308dc","Type":"ContainerDied","Data":"5c46e787c94fffad6300813162958e1924fc284bced21adfb9c810b5b661d5ce"}
Nov 01 08:37:45.247364 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.246903    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c46e787c94fffad6300813162958e1924fc284bced21adfb9c810b5b661d5ce"
Nov 01 08:37:45.249163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.249137    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9" event={"ID":"045647ae-d63f-4d63-9188-b624fa59f605","Type":"ContainerDied","Data":"d6af6860b487dc902e54b39f399408cde1d0b0bf6da6faca9b07e78e621d92c0"}
Nov 01 08:37:45.249273 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.249182    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6af6860b487dc902e54b39f399408cde1d0b0bf6da6faca9b07e78e621d92c0"
Nov 01 08:37:45.249273 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.249217    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbzn9"
Nov 01 08:37:45.385401 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.385378    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:45.387373 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.387358    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:45.444429 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.444409    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util\") pod \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") "
Nov 01 08:37:45.444548 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.444444    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle\") pod \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") "
Nov 01 08:37:45.444548 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.444471    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp\" (UniqueName: \"kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp\") pod \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\" (UID: \"cc612aa0-e69c-4ad4-8b03-af18a7841d5e\") "
Nov 01 08:37:45.444548 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.444527    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr\" (UniqueName: \"kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr\") pod \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") "
Nov 01 08:37:45.444709 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.444584    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle\") pod \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") "
Nov 01 08:37:45.444709 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.444618    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util\") pod \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\" (UID: \"95ccaed6-96d7-4be0-accf-b00784a7d6e2\") "
Nov 01 08:37:45.445073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.445046    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle" pod "cc612aa0-e69c-4ad4-8b03-af18a7841d5e" (UID: "cc612aa0-e69c-4ad4-8b03-af18a7841d5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:45.445185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.445144    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle" pod "95ccaed6-96d7-4be0-accf-b00784a7d6e2" (UID: "95ccaed6-96d7-4be0-accf-b00784a7d6e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:45.446650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.446619    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr" pod "95ccaed6-96d7-4be0-accf-b00784a7d6e2" (UID: "95ccaed6-96d7-4be0-accf-b00784a7d6e2"). InnerVolumeSpecName "kube-api-access-xwdmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:45.446745 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.446707    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp" pod "cc612aa0-e69c-4ad4-8b03-af18a7841d5e" (UID: "cc612aa0-e69c-4ad4-8b03-af18a7841d5e"). InnerVolumeSpecName "kube-api-access-qhrgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:45.450852 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.450824    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util" pod "95ccaed6-96d7-4be0-accf-b00784a7d6e2" (UID: "95ccaed6-96d7-4be0-accf-b00784a7d6e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:45.452464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.452439    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util" pod "cc612aa0-e69c-4ad4-8b03-af18a7841d5e" (UID: "cc612aa0-e69c-4ad4-8b03-af18a7841d5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:45.545104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.545044    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdmr\" (UniqueName: \"kubernetes.io/projected/95ccaed6-96d7-4be0-accf-b00784a7d6e2-kube-api-access-xwdmr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:45.545104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.545078    4314 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:45.545104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.545088    4314 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95ccaed6-96d7-4be0-accf-b00784a7d6e2-util\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:45.545104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.545096    4314 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-util\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:45.545104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.545104    4314 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:45.545104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:45.545113    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrgp\" (UniqueName: \"kubernetes.io/projected/cc612aa0-e69c-4ad4-8b03-af18a7841d5e-kube-api-access-qhrgp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:46.254984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:46.254954    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p"
Nov 01 08:37:46.254984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:46.254967    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qnj2p" event={"ID":"95ccaed6-96d7-4be0-accf-b00784a7d6e2","Type":"ContainerDied","Data":"ea7ffd3f5d754be6460061ae14bf6f3bc8189974247074608cafd4109e59f3f8"}
Nov 01 08:37:46.255458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:46.255008    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea7ffd3f5d754be6460061ae14bf6f3bc8189974247074608cafd4109e59f3f8"
Nov 01 08:37:46.256902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:46.256865    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn" event={"ID":"cc612aa0-e69c-4ad4-8b03-af18a7841d5e","Type":"ContainerDied","Data":"c46e49312a7f5d981d4c6e0493fc069a7a76ffa66cf7e126a33ca1cdfdf14231"}
Nov 01 08:37:46.256902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:46.256897    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c672cpn"
Nov 01 08:37:46.257016 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:46.256910    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46e49312a7f5d981d4c6e0493fc069a7a76ffa66cf7e126a33ca1cdfdf14231"
Nov 01 08:37:53.618091 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:53.618054    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:53.767839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:53.767810    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:54.483884 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.483828    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:54.729547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729509    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"]
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729781    4314 cpu_manager.go:414] "containerMap: removing container" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729792    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729803    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729808    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729816    4314 cpu_manager.go:414] "containerMap: removing container" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729821    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729828    4314 cpu_manager.go:414] "containerMap: removing container" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729832    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729844    4314 cpu_manager.go:414] "containerMap: removing container" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729848    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729854    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729858    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729865    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729869    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729877    4314 cpu_manager.go:414] "containerMap: removing container" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729882    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729892    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729896    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="pull"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729904    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729908    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="util"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729912    4314 cpu_manager.go:414] "containerMap: removing container" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729916    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729921    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="extract"
Nov 01 08:37:54.729934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729925    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="extract"
Nov 01 08:37:54.730724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729971    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="045647ae-d63f-4d63-9188-b624fa59f605" containerName="extract"
Nov 01 08:37:54.730724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729977    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc612aa0-e69c-4ad4-8b03-af18a7841d5e" containerName="extract"
Nov 01 08:37:54.730724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729984    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ccaed6-96d7-4be0-accf-b00784a7d6e2" containerName="extract"
Nov 01 08:37:54.730724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.729990    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1bac9f9-cb0e-4388-a9f3-f68f926308dc" containerName="extract"
Nov 01 08:37:54.732942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.732927    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.781490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.781440    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 08:37:54.782026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.781991    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gx5kp\""
Nov 01 08:37:54.782145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.782032    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 08:37:54.782145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.782035    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 08:37:54.782145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.782059    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 08:37:54.782145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.782067    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 08:37:54.814242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.814218    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.814346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.814251    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.814346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.814284    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.814422 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.814381    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.814422 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.814403    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdfl\" (UniqueName: \"kubernetes.io/projected/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-kube-api-access-lzdfl\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.814496 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.814442    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.822877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.822847    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:54.860828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.860805    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"]
Nov 01 08:37:54.915257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915230    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.915379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915272    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.915379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915299    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.915449 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915409    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdfl\" (UniqueName: \"kubernetes.io/projected/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-kube-api-access-lzdfl\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.915490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915468    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.915553 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915527    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.915942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.915909    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.917698 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.917674    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.917807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.917674    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.917807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.917771    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.917886 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.917805    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.972241 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.972214    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdfl\" (UniqueName: \"kubernetes.io/projected/d7ecf794-d1ab-4d30-ad0a-05c7675a448e-kube-api-access-lzdfl\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-gfx67\" (UID: \"d7ecf794-d1ab-4d30-ad0a-05c7675a448e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:54.975316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:54.975288    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:55.047027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:55.046974    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:37:55.122105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:55.122078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:55.520215 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:55.520191    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"]
Nov 01 08:37:55.522184 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:37:55.522156    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ecf794_d1ab_4d30_ad0a_05c7675a448e.slice/crio-51b631fed6432b72e8a701c282006cbb89fd52e2163ffe5040b35e5ccbc7f75d WatchSource:0}: Error finding container 51b631fed6432b72e8a701c282006cbb89fd52e2163ffe5040b35e5ccbc7f75d: Status 404 returned error can't find the container with id 51b631fed6432b72e8a701c282006cbb89fd52e2163ffe5040b35e5ccbc7f75d
Nov 01 08:37:56.296965 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:56.296926    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67" event={"ID":"d7ecf794-d1ab-4d30-ad0a-05c7675a448e","Type":"ContainerStarted","Data":"51b631fed6432b72e8a701c282006cbb89fd52e2163ffe5040b35e5ccbc7f75d"}
Nov 01 08:37:57.397966 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:57.397900    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:58.221182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:58.221141    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:58.595121 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:58.595074    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:37:59.313667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:59.313623    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67" event={"ID":"d7ecf794-d1ab-4d30-ad0a-05c7675a448e","Type":"ContainerStarted","Data":"22708f3ccd5058dbe831907ce4c27a04fe44e15740c4ea6c98d72d08bde6cbef"}
Nov 01 08:37:59.839729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:37:59.839652    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:00.292725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:00.292691    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:00.319240 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:00.319208    4314 generic.go:356] "Generic (PLEG): container finished" podID="d7ecf794-d1ab-4d30-ad0a-05c7675a448e" containerID="22708f3ccd5058dbe831907ce4c27a04fe44e15740c4ea6c98d72d08bde6cbef" exitCode=0
Nov 01 08:38:00.319344 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:00.319289    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67" event={"ID":"d7ecf794-d1ab-4d30-ad0a-05c7675a448e","Type":"ContainerDied","Data":"22708f3ccd5058dbe831907ce4c27a04fe44e15740c4ea6c98d72d08bde6cbef"}
Nov 01 08:38:01.324765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:01.324730    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67" event={"ID":"d7ecf794-d1ab-4d30-ad0a-05c7675a448e","Type":"ContainerStarted","Data":"a63e5d977995494e5428b19ae422021a6607a14da9a9051e911719175bdd6c1e"}
Nov 01 08:38:01.325082 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:01.324820    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:38:01.547937 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:01.547828    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67" podStartSLOduration=2.638708741 podStartE2EDuration="7.547813538s" totalImagesPullingTime="4.47879161s" totalInitContainerRuntime="430.313187ms" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:01.500923078 +0000 UTC m=+354.695317623" watchObservedRunningTime="2026-11-01 08:38:01.547813538 +0000 UTC m=+354.742208090"
Nov 01 08:38:01.692209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:01.692164    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:01.844841 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:01.844731    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:04.509257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.509223    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"]
Nov 01 08:38:04.515610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.515592    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.567415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.567394    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"]
Nov 01 08:38:04.590217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.590195    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sw68\" (UniqueName: \"kubernetes.io/projected/a7d4d063-11e7-4e3e-93fb-2a47424dcc21-kube-api-access-2sw68\") pod \"jobset-operator-674cd59dc5-hh9zl\" (UID: \"a7d4d063-11e7-4e3e-93fb-2a47424dcc21\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.590339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.590237    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d4d063-11e7-4e3e-93fb-2a47424dcc21-tmp\") pod \"jobset-operator-674cd59dc5-hh9zl\" (UID: \"a7d4d063-11e7-4e3e-93fb-2a47424dcc21\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.611967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.611941    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:38:04.612076 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.611995    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 08:38:04.612076 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.612070    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-bvgjq\""
Nov 01 08:38:04.691047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.691014    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sw68\" (UniqueName: \"kubernetes.io/projected/a7d4d063-11e7-4e3e-93fb-2a47424dcc21-kube-api-access-2sw68\") pod \"jobset-operator-674cd59dc5-hh9zl\" (UID: \"a7d4d063-11e7-4e3e-93fb-2a47424dcc21\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.691195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.691067    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d4d063-11e7-4e3e-93fb-2a47424dcc21-tmp\") pod \"jobset-operator-674cd59dc5-hh9zl\" (UID: \"a7d4d063-11e7-4e3e-93fb-2a47424dcc21\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.691435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.691413    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d4d063-11e7-4e3e-93fb-2a47424dcc21-tmp\") pod \"jobset-operator-674cd59dc5-hh9zl\" (UID: \"a7d4d063-11e7-4e3e-93fb-2a47424dcc21\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.748326 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.748299    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sw68\" (UniqueName: \"kubernetes.io/projected/a7d4d063-11e7-4e3e-93fb-2a47424dcc21-kube-api-access-2sw68\") pod \"jobset-operator-674cd59dc5-hh9zl\" (UID: \"a7d4d063-11e7-4e3e-93fb-2a47424dcc21\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:04.833925 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:04.833847    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"
Nov 01 08:38:05.195214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:05.195185    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl"]
Nov 01 08:38:05.197698 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:38:05.197670    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7d4d063_11e7_4e3e_93fb_2a47424dcc21.slice/crio-309d2561a6f09a2510ca4fd25c9ed8826a64f073ac8845de9e19f4ed5f574c9d WatchSource:0}: Error finding container 309d2561a6f09a2510ca4fd25c9ed8826a64f073ac8845de9e19f4ed5f574c9d: Status 404 returned error can't find the container with id 309d2561a6f09a2510ca4fd25c9ed8826a64f073ac8845de9e19f4ed5f574c9d
Nov 01 08:38:05.339491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:05.339464    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl" event={"ID":"a7d4d063-11e7-4e3e-93fb-2a47424dcc21","Type":"ContainerStarted","Data":"309d2561a6f09a2510ca4fd25c9ed8826a64f073ac8845de9e19f4ed5f574c9d"}
Nov 01 08:38:05.667241 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:05.667202    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:05.824887 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:05.824854    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:07.203392 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:07.203354    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:08.570765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:08.570724    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:08.735991 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:08.735958    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:08.889876 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:08.889790    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:10.363885 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:10.363851    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl" event={"ID":"a7d4d063-11e7-4e3e-93fb-2a47424dcc21","Type":"ContainerStarted","Data":"4c1cd87b06e12a5494c9dc05d04b721a31541a3e24ecc7b1d4e2604ed0d25e7c"}
Nov 01 08:38:10.475339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:10.475284    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hh9zl" podStartSLOduration=1.998276478 podStartE2EDuration="6.47527065s" totalImagesPullingTime="4.476994172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:10.473825314 +0000 UTC m=+363.668219870" watchObservedRunningTime="2026-11-01 08:38:10.47527065 +0000 UTC m=+363.669665204"
Nov 01 08:38:10.794510 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:10.794476    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:11.321173 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:38:11.321136    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:38:11.435619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:11.435592    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:11.590947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:11.590865    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:12.331122 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:12.331089    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-gfx67"
Nov 01 08:38:13.660680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:13.660633    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:15.045181 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:15.045152    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:16.442956 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:16.442921    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:17.429399 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:17.429367    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:18.308330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.308292    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:18.332022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.331995    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"]
Nov 01 08:38:18.334983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.334961    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.383331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.383300    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 01 08:38:18.429379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.429355    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wx6gn\""
Nov 01 08:38:18.429379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.429374    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 08:38:18.429379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.429381    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 08:38:18.429704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.429394    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 01 08:38:18.430928 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.430906    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"]
Nov 01 08:38:18.510185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.510148    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.510350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.510195    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldfs\" (UniqueName: \"kubernetes.io/projected/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-kube-api-access-sldfs\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.510350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.510258    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.558156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.558131    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:18.611689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.611619    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.611689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.611673    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldfs\" (UniqueName: \"kubernetes.io/projected/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-kube-api-access-sldfs\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.611832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.611702    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.612277 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.612254    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.614086 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.614065    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.668591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.668559    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldfs\" (UniqueName: \"kubernetes.io/projected/f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad-kube-api-access-sldfs\") pod \"kuadrant-console-plugin-66c4fb575d-h6b62\" (UID: \"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:18.944471 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:18.944396    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"
Nov 01 08:38:19.358470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.358383    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62"]
Nov 01 08:38:19.361068 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:38:19.361042    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf96fe7be_b8d7_4c5d_bbbe_dd5b8143a4ad.slice/crio-67dafb6373bec1e86430ccb6ea52bed6df81edc5c3f0ef7118ebebf9b73ad7c0 WatchSource:0}: Error finding container 67dafb6373bec1e86430ccb6ea52bed6df81edc5c3f0ef7118ebebf9b73ad7c0: Status 404 returned error can't find the container with id 67dafb6373bec1e86430ccb6ea52bed6df81edc5c3f0ef7118ebebf9b73ad7c0
Nov 01 08:38:19.403385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.403347    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62" event={"ID":"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad","Type":"ContainerStarted","Data":"67dafb6373bec1e86430ccb6ea52bed6df81edc5c3f0ef7118ebebf9b73ad7c0"}
Nov 01 08:38:19.743095 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.743061    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"]
Nov 01 08:38:19.747430 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.747402    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:19.838717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.838675    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:19.845864 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.845839    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hpdgl\""
Nov 01 08:38:19.847347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.847325    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"]
Nov 01 08:38:19.923752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:19.923719    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwhr\" (UniqueName: \"kubernetes.io/projected/f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e-kube-api-access-xtwhr\") pod \"limitador-operator-controller-manager-5778bb4449-l9ctz\" (UID: \"f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:20.024807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:20.024712    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwhr\" (UniqueName: \"kubernetes.io/projected/f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e-kube-api-access-xtwhr\") pod \"limitador-operator-controller-manager-5778bb4449-l9ctz\" (UID: \"f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:20.082023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:20.081995    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwhr\" (UniqueName: \"kubernetes.io/projected/f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e-kube-api-access-xtwhr\") pod \"limitador-operator-controller-manager-5778bb4449-l9ctz\" (UID: \"f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:20.360352 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:20.360265    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:20.728526 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:20.728472    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"]
Nov 01 08:38:20.731236 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:38:20.731211    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0de96e5_dcbb_4a01_a4a9_10d3b4ec230e.slice/crio-046af5dfbda01803493305b212e6e315f1289b4cf079379557b42330e9a34859 WatchSource:0}: Error finding container 046af5dfbda01803493305b212e6e315f1289b4cf079379557b42330e9a34859: Status 404 returned error can't find the container with id 046af5dfbda01803493305b212e6e315f1289b4cf079379557b42330e9a34859
Nov 01 08:38:21.413365 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:21.413331    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz" event={"ID":"f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e","Type":"ContainerStarted","Data":"046af5dfbda01803493305b212e6e315f1289b4cf079379557b42330e9a34859"}
Nov 01 08:38:23.202153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:23.202112    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:23.352683 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:23.352648    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:23.571425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:23.571388    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:23.754122 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:23.754089    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:23.979748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:23.979657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:24.408812 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:24.408780    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:24.562302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:24.562259    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:25.450480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:25.450441    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:25.628528 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:25.628488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:27.441258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:27.441219    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz" event={"ID":"f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e","Type":"ContainerStarted","Data":"73234a021e1d0e9787f4e19bcda18ea7773d24b852237188a728e2822e74a838"}
Nov 01 08:38:27.441662 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:27.441364    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:27.551014 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:27.550955    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz" podStartSLOduration=2.6409644610000003 podStartE2EDuration="8.550941631s" totalImagesPullingTime="5.90997717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:27.549661615 +0000 UTC m=+380.744056170" watchObservedRunningTime="2026-11-01 08:38:27.550941631 +0000 UTC m=+380.745336184"
Nov 01 08:38:27.821554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:27.821504    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:28.290522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.290486    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"]
Nov 01 08:38:28.293798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.293782    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.298650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.298625    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:28.306357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.306334    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f9a5a60-c399-441a-b22d-1ab01e2959f0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r6x45\" (UID: \"7f9a5a60-c399-441a-b22d-1ab01e2959f0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.306442 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.306363    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9gn\" (UniqueName: \"kubernetes.io/projected/7f9a5a60-c399-441a-b22d-1ab01e2959f0-kube-api-access-cs9gn\") pod \"cert-manager-cainjector-5c57bbd4fc-r6x45\" (UID: \"7f9a5a60-c399-441a-b22d-1ab01e2959f0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.348751 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.348719    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 08:38:28.348751 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.348748    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 08:38:28.350182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.350156    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"]
Nov 01 08:38:28.388546 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.388502    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l28bn\""
Nov 01 08:38:28.407812 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.407779    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f9a5a60-c399-441a-b22d-1ab01e2959f0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r6x45\" (UID: \"7f9a5a60-c399-441a-b22d-1ab01e2959f0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.407962 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.407823    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9gn\" (UniqueName: \"kubernetes.io/projected/7f9a5a60-c399-441a-b22d-1ab01e2959f0-kube-api-access-cs9gn\") pod \"cert-manager-cainjector-5c57bbd4fc-r6x45\" (UID: \"7f9a5a60-c399-441a-b22d-1ab01e2959f0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.466067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.466025    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f9a5a60-c399-441a-b22d-1ab01e2959f0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r6x45\" (UID: \"7f9a5a60-c399-441a-b22d-1ab01e2959f0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.466486 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.466179    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9gn\" (UniqueName: \"kubernetes.io/projected/7f9a5a60-c399-441a-b22d-1ab01e2959f0-kube-api-access-cs9gn\") pod \"cert-manager-cainjector-5c57bbd4fc-r6x45\" (UID: \"7f9a5a60-c399-441a-b22d-1ab01e2959f0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.605090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.605010    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"
Nov 01 08:38:28.969704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:28.969677    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45"]
Nov 01 08:38:28.971701 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:38:28.971672    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9a5a60_c399_441a_b22d_1ab01e2959f0.slice/crio-f73185945774f3e13f8b0e6470c680cc404015edb795582bd268e9c5f4cf4491 WatchSource:0}: Error finding container f73185945774f3e13f8b0e6470c680cc404015edb795582bd268e9c5f4cf4491: Status 404 returned error can't find the container with id f73185945774f3e13f8b0e6470c680cc404015edb795582bd268e9c5f4cf4491
Nov 01 08:38:29.451248 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:29.451206    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45" event={"ID":"7f9a5a60-c399-441a-b22d-1ab01e2959f0","Type":"ContainerStarted","Data":"f73185945774f3e13f8b0e6470c680cc404015edb795582bd268e9c5f4cf4491"}
Nov 01 08:38:31.510457 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:31.510359    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:31.661158 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:31.661107    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:32.010090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:32.010051    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:32.804124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:32.804019    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:32.952848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:32.952814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:33.369814 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.369775    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:33.422037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.422008    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"]
Nov 01 08:38:33.425704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.425683    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.448551 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.448506    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d3a6367-0131-48e9-b825-19cf48b23b87-tmp\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.448651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.448614    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3a6367-0131-48e9-b825-19cf48b23b87-tls-certs\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.448702 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.448668    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tsz\" (UniqueName: \"kubernetes.io/projected/2d3a6367-0131-48e9-b825-19cf48b23b87-kube-api-access-65tsz\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.478477 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.478441    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62" event={"ID":"f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad","Type":"ContainerStarted","Data":"a260f2cba82e01a78e30b0b5d4f443a0285aac609beb3ad71dd3431dadd7cb19"}
Nov 01 08:38:33.523022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.522990    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 08:38:33.523188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.523062    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 08:38:33.523188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.523169    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 08:38:33.524916 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.524893    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"]
Nov 01 08:38:33.525113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.525096    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rpqnp\""
Nov 01 08:38:33.525113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.525107    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 08:38:33.549923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.549559    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d3a6367-0131-48e9-b825-19cf48b23b87-tmp\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.549923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.549610    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3a6367-0131-48e9-b825-19cf48b23b87-tls-certs\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.549923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.549653    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tsz\" (UniqueName: \"kubernetes.io/projected/2d3a6367-0131-48e9-b825-19cf48b23b87-kube-api-access-65tsz\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.552380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.552351    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d3a6367-0131-48e9-b825-19cf48b23b87-tmp\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.552612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.552590    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3a6367-0131-48e9-b825-19cf48b23b87-tls-certs\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.619019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.618985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:33.621173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.621106    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tsz\" (UniqueName: \"kubernetes.io/projected/2d3a6367-0131-48e9-b825-19cf48b23b87-kube-api-access-65tsz\") pod \"kube-auth-proxy-5649b5c55d-mck42\" (UID: \"2d3a6367-0131-48e9-b825-19cf48b23b87\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:33.677555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.677478    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h6b62" podStartSLOduration=2.540043768 podStartE2EDuration="15.67744815s" totalImagesPullingTime="13.137404382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:33.67690964 +0000 UTC m=+386.871304189" watchObservedRunningTime="2026-11-01 08:38:33.67744815 +0000 UTC m=+386.871842705"
Nov 01 08:38:33.739727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:33.739704    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"
Nov 01 08:38:34.063892 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:34.063856    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:34.115885 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:34.115566    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5649b5c55d-mck42"]
Nov 01 08:38:34.118076 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:38:34.118046    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3a6367_0131_48e9_b825_19cf48b23b87.slice/crio-21da92318fec1ca69cd54b8f69a364ead8a3f9b3252b69d3cc99c142ebb66c95 WatchSource:0}: Error finding container 21da92318fec1ca69cd54b8f69a364ead8a3f9b3252b69d3cc99c142ebb66c95: Status 404 returned error can't find the container with id 21da92318fec1ca69cd54b8f69a364ead8a3f9b3252b69d3cc99c142ebb66c95
Nov 01 08:38:34.231380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:34.231327    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:34.383780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:34.383669    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:34.484839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:34.484799    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45" event={"ID":"7f9a5a60-c399-441a-b22d-1ab01e2959f0","Type":"ContainerStarted","Data":"60df47308f3c83bce58592e1372e8bc7a450f010c5ca457d4e97c9d3dd6204c2"}
Nov 01 08:38:34.486116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:34.486093    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42" event={"ID":"2d3a6367-0131-48e9-b825-19cf48b23b87","Type":"ContainerStarted","Data":"21da92318fec1ca69cd54b8f69a364ead8a3f9b3252b69d3cc99c142ebb66c95"}
Nov 01 08:38:36.200390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:36.200357    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:37.834496 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:37.834435    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:37.981981 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:37.981945    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:38.448104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:38.448065    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-l9ctz"
Nov 01 08:38:38.560581 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:38.560498    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r6x45" podStartSLOduration=5.809584042 podStartE2EDuration="10.560480059s" totalImagesPullingTime="4.750896017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:34.592710897 +0000 UTC m=+387.787105452" watchObservedRunningTime="2026-11-01 08:38:38.560480059 +0000 UTC m=+391.754874616"
Nov 01 08:38:38.965643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:38.965608    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:39.135515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:39.135430    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:39.287807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:39.287774    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:39.510139 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:39.510107    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42" event={"ID":"2d3a6367-0131-48e9-b825-19cf48b23b87","Type":"ContainerStarted","Data":"28b90d3187be7584284732c3aff3aaa4711a4ac252ee2135cf9ae07a25eac278"}
Nov 01 08:38:39.623178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:39.623128    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-mck42" podStartSLOduration=1.975765158 podStartE2EDuration="6.623114642s" totalImagesPullingTime="4.647349484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:39.621273798 +0000 UTC m=+392.815668366" watchObservedRunningTime="2026-11-01 08:38:39.623114642 +0000 UTC m=+392.817509209"
Nov 01 08:38:40.435498 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:40.435466    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:40.574806 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:40.574775    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:40.814333 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:40.814302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:40.959608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:40.959569    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:41.320874 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:38:41.320840    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:38:45.656262 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:45.656211    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:45.796948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:45.796917    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:52.143019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:52.142946    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:52.286260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:52.286230    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:58.453758 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.453723    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"]
Nov 01 08:38:58.461586 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.461570    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.510294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.510265    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 08:38:58.510294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.510280    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 08:38:58.547721 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.547690    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f09aaaf-f882-4e11-b764-bf87a9351d3e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.547846 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.547747    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f09aaaf-f882-4e11-b764-bf87a9351d3e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.547846 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.547764    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndbc\" (UniqueName: \"kubernetes.io/projected/9f09aaaf-f882-4e11-b764-bf87a9351d3e-kube-api-access-wndbc\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.547920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.547844    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f09aaaf-f882-4e11-b764-bf87a9351d3e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.556460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.556437    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-8ql8w\""
Nov 01 08:38:58.648979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.648944    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f09aaaf-f882-4e11-b764-bf87a9351d3e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.648979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.648987    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndbc\" (UniqueName: \"kubernetes.io/projected/9f09aaaf-f882-4e11-b764-bf87a9351d3e-kube-api-access-wndbc\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.649197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.649019    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f09aaaf-f882-4e11-b764-bf87a9351d3e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.649197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.649083    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f09aaaf-f882-4e11-b764-bf87a9351d3e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.649405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.649385    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f09aaaf-f882-4e11-b764-bf87a9351d3e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.649463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.649414    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f09aaaf-f882-4e11-b764-bf87a9351d3e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.649500 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.649460    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 08:38:58.651229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.651207    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"]
Nov 01 08:38:58.660571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.660524    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f09aaaf-f882-4e11-b764-bf87a9351d3e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.701387 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.701365    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:58.706129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.706083    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndbc\" (UniqueName: \"kubernetes.io/projected/9f09aaaf-f882-4e11-b764-bf87a9351d3e-kube-api-access-wndbc\") pod \"kserve-module-controller-manager-7566cf5fc8-8jqs6\" (UID: \"9f09aaaf-f882-4e11-b764-bf87a9351d3e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.774201 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.774180    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:38:58.845227 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:58.845151    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:38:59.189375 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:59.189252    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"]
Nov 01 08:38:59.191921 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:38:59.191897    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f09aaaf_f882_4e11_b764_bf87a9351d3e.slice/crio-00f05c592363185b7b13cdb651451b0b5d989b6cbd7df33f7dd59d81619d2358 WatchSource:0}: Error finding container 00f05c592363185b7b13cdb651451b0b5d989b6cbd7df33f7dd59d81619d2358: Status 404 returned error can't find the container with id 00f05c592363185b7b13cdb651451b0b5d989b6cbd7df33f7dd59d81619d2358
Nov 01 08:38:59.598061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:59.598031    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6" event={"ID":"9f09aaaf-f882-4e11-b764-bf87a9351d3e","Type":"ContainerStarted","Data":"00f05c592363185b7b13cdb651451b0b5d989b6cbd7df33f7dd59d81619d2358"}
Nov 01 08:38:59.990577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:38:59.990517    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:01.608250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:01.608181    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6" event={"ID":"9f09aaaf-f882-4e11-b764-bf87a9351d3e","Type":"ContainerStarted","Data":"d79954d7c97ba5dbcb620e60008960fd5b3e146a23a5603926ab3a9b58876183"}
Nov 01 08:39:01.608648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:01.608385    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:39:01.766427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:01.766371    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6" podStartSLOduration=1.891668865 podStartE2EDuration="3.766356448s" totalImagesPullingTime="1.874687583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:01.764964622 +0000 UTC m=+414.959359178" watchObservedRunningTime="2026-11-01 08:39:01.766356448 +0000 UTC m=+414.960751001"
Nov 01 08:39:02.021414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:02.021383    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:02.307499 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:02.307467    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:02.451842 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:02.451800    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:08.699599 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:08.699565    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:08.841581 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:08.841548    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:09.307423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:09.307392    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:09.457406 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:09.457348    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:09.898502 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:09.898471    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:10.038925 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:10.038899    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:11.321150 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:39:11.321115    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:39:14.618505 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:14.618472    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:14.759919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:14.759893    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:15.832089 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:15.832057    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:15.975715 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:15.975678    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:16.117185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:16.117103    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:17.366613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.366577    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"]
Nov 01 08:39:17.396725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.396693    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.500501 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.500461    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fafe313-fdae-45f3-9133-63465e3f725c-cert\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.500678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.500570    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fafe313-fdae-45f3-9133-63465e3f725c-metrics-certs\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.500678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.500638    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fafe313-fdae-45f3-9133-63465e3f725c-manager-config\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.500802 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.500755    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7mr\" (UniqueName: \"kubernetes.io/projected/1fafe313-fdae-45f3-9133-63465e3f725c-kube-api-access-fp7mr\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.503564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.503523    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 08:39:17.503692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.503650    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 08:39:17.503800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.503781    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-hvsx6\""
Nov 01 08:39:17.503854 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.503786    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 08:39:17.506220 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.506198    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"]
Nov 01 08:39:17.601121 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.601094    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fafe313-fdae-45f3-9133-63465e3f725c-cert\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.601258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.601125    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fafe313-fdae-45f3-9133-63465e3f725c-metrics-certs\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.601258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.601163    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fafe313-fdae-45f3-9133-63465e3f725c-manager-config\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.601258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.601215    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7mr\" (UniqueName: \"kubernetes.io/projected/1fafe313-fdae-45f3-9133-63465e3f725c-kube-api-access-fp7mr\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.601912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.601889    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fafe313-fdae-45f3-9133-63465e3f725c-manager-config\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.603578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.603550    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fafe313-fdae-45f3-9133-63465e3f725c-metrics-certs\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.603676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.603585    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fafe313-fdae-45f3-9133-63465e3f725c-cert\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.658148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.658091    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7mr\" (UniqueName: \"kubernetes.io/projected/1fafe313-fdae-45f3-9133-63465e3f725c-kube-api-access-fp7mr\") pod \"jobset-controller-manager-784b9f44f5-w9pff\" (UID: \"1fafe313-fdae-45f3-9133-63465e3f725c\") " pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:17.708912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:17.708887    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:18.070208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:18.070139    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"]
Nov 01 08:39:18.072411 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:39:18.072378    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fafe313_fdae_45f3_9133_63465e3f725c.slice/crio-cb147995e1a6a1ccfb401d947888343de506f8064cfac10ec52437f70bc5b06f WatchSource:0}: Error finding container cb147995e1a6a1ccfb401d947888343de506f8064cfac10ec52437f70bc5b06f: Status 404 returned error can't find the container with id cb147995e1a6a1ccfb401d947888343de506f8064cfac10ec52437f70bc5b06f
Nov 01 08:39:18.681747 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:18.681715    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff" event={"ID":"1fafe313-fdae-45f3-9133-63465e3f725c","Type":"ContainerStarted","Data":"cb147995e1a6a1ccfb401d947888343de506f8064cfac10ec52437f70bc5b06f"}
Nov 01 08:39:21.168609 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:21.168565    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:21.311376 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:21.311343    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:22.614052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:22.614023    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8jqs6"
Nov 01 08:39:23.705968 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:23.705931    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff" event={"ID":"1fafe313-fdae-45f3-9133-63465e3f725c","Type":"ContainerStarted","Data":"76e41dc28d2f0346a7d87e2105b36ded9bfa1a492a4a2f2d82707b9a9b22788c"}
Nov 01 08:39:23.706335 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:23.706106    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:23.815178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:23.815135    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff" podStartSLOduration=2.233528859 podStartE2EDuration="6.81512166s" totalImagesPullingTime="4.581592801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:23.814691778 +0000 UTC m=+437.009086326" watchObservedRunningTime="2026-11-01 08:39:23.81512166 +0000 UTC m=+437.009516277"
Nov 01 08:39:24.172779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:24.172745    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:24.312178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:24.312146    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:24.828771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:24.828739    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:24.971755 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:24.971721    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:25.521705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:25.521674    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:32.386218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:32.386141    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:32.525482 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:32.525452    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:34.714819 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:34.714791    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-784b9f44f5-w9pff"
Nov 01 08:39:41.320654 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:39:41.320611    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:39:47.206490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:47.206455    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:47.355084 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:47.355050    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:47.498692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:47.498631    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:47.639435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:47.639401    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:56.916700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:56.916663    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:39:57.071034 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:39:57.070995    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:01.648493 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:01.648463    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:01.790633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:01.790601    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:02.376870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:02.376835    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:02.681717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:02.681641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:02.825602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:02.825572    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:02.966622 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:02.966526    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:03.112957 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:03.112924    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:03.257734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:03.257701    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:03.400391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:03.400365    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:03.541595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:03.541503    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:03.686703 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:03.686668    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:04.776167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:04.776106    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:09.516408 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:09.516367    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:09.661342 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:09.661303    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:09.800624 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:09.800527    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:09.940543 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:09.940509    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:10.088840 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:10.088771    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:10.231209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:10.231182    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:10.379327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:10.379255    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:10.519728 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:10.519702    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:10.662776 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:10.662706    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:11.321621 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:40:11.321583    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:40:27.891496 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:27.891421    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:28.036758 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:28.036725    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:30.003789 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:30.003759    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:30.168979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:30.168944    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:30.383801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:30.383732    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:41.321145 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:40:41.321109    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:40:45.561911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:45.561872    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:40:45.754904 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:40:45.754868    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:02.716701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:02.716661    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:11.321295 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:41:11.321252    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:41:14.011978 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:14.011934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:41.321914 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:41:41.321815    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:41:42.984129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:42.984091    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:45.133414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:45.133379    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:45.306119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:45.306083    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:51.747074 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:51.746996    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:41:51.907999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:41:51.907943    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:03.060028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:03.059989    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:11.322386 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:42:11.322348    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:42:14.215221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:14.215187    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:14.668317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:14.668280    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:15.873165 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:15.873132    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:16.013641 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:16.013604    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:16.233765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:16.233731    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:22.544572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:22.544509    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:31.300952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:31.300909    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:34.356169 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:34.356117    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:37.386133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.386102    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"]
Nov 01 08:42:37.389434 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.389416    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.438381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.438348    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 08:42:37.464117 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.464083    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3855ed68-9c33-4e39-b69a-c56f27fa6d46-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.464293 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.464139    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.464293 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.464164    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.464293 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.464223    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.464293 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.464250    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dhk\" (UniqueName: \"kubernetes.io/projected/3855ed68-9c33-4e39-b69a-c56f27fa6d46-kube-api-access-86dhk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.464458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.464423    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.477979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.477944    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:37.484163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.484142    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 08:42:37.484374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.484357    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 08:42:37.484454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.484441    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ffzjt\""
Nov 01 08:42:37.486313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.486291    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"]
Nov 01 08:42:37.565417 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565378    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dhk\" (UniqueName: \"kubernetes.io/projected/3855ed68-9c33-4e39-b69a-c56f27fa6d46-kube-api-access-86dhk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.565649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565448    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.565649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565494    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3855ed68-9c33-4e39-b69a-c56f27fa6d46-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.565649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565569    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.565826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565690    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.565826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565732    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.565949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.565925    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.566043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.566023    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.566084 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.566051    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.567841 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.567815    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3855ed68-9c33-4e39-b69a-c56f27fa6d46-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.568109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.568094    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3855ed68-9c33-4e39-b69a-c56f27fa6d46-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.622762 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.622724    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dhk\" (UniqueName: \"kubernetes.io/projected/3855ed68-9c33-4e39-b69a-c56f27fa6d46-kube-api-access-86dhk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xcr5m\" (UID: \"3855ed68-9c33-4e39-b69a-c56f27fa6d46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:37.703552 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:37.703446    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:38.068512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:38.068483    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"]
Nov 01 08:42:38.070435 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:42:38.070411    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3855ed68_9c33_4e39_b69a_c56f27fa6d46.slice/crio-3b546cf3fde9d230298542a906d6f3ed0b10232fe6bef35ecd767217930348f5 WatchSource:0}: Error finding container 3b546cf3fde9d230298542a906d6f3ed0b10232fe6bef35ecd767217930348f5: Status 404 returned error can't find the container with id 3b546cf3fde9d230298542a906d6f3ed0b10232fe6bef35ecd767217930348f5
Nov 01 08:42:38.071848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:38.071822    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:42:38.539892 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:38.539856    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m" event={"ID":"3855ed68-9c33-4e39-b69a-c56f27fa6d46","Type":"ContainerStarted","Data":"3b546cf3fde9d230298542a906d6f3ed0b10232fe6bef35ecd767217930348f5"}
Nov 01 08:42:40.550875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:40.550834    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m" event={"ID":"3855ed68-9c33-4e39-b69a-c56f27fa6d46","Type":"ContainerStarted","Data":"c684905139ad0963988c545ac0e12367e09695f637ba033e72f929490adca80e"}
Nov 01 08:42:40.662229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:40.662166    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m" podStartSLOduration=2.165061738 podStartE2EDuration="3.662153475s" totalImagesPullingTime="1.497091737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:40.660527562 +0000 UTC m=+633.854922117" watchObservedRunningTime="2026-11-01 08:42:40.662153475 +0000 UTC m=+633.856548029"
Nov 01 08:42:40.697676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:40.697639    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:40.704522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:40.704500    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:41.323012 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:42:41.322973    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:42:44.467422 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.467389    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-866l2"]
Nov 01 08:42:44.470832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.470815    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.520861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.520829    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3d132e-8479-4c24-a5a1-8dcc1d404880-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.521037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.520871    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.521037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.520940    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.521037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.520992    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt7x\" (UniqueName: \"kubernetes.io/projected/cc3d132e-8479-4c24-a5a1-8dcc1d404880-kube-api-access-2tt7x\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.521208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.521100    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-home\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.521208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.521134    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.557696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.557658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:44.565380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.565355    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 08:42:44.567873 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.567853    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-866l2"]
Nov 01 08:42:44.622603 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.622562    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3d132e-8479-4c24-a5a1-8dcc1d404880-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.622813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.622620    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.622813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.622653    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.622813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.622674    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt7x\" (UniqueName: \"kubernetes.io/projected/cc3d132e-8479-4c24-a5a1-8dcc1d404880-kube-api-access-2tt7x\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.622813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.622724    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-home\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.622813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.622746    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.623175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.623145    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-home\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.623248 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.623160    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.623248 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.623204    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.624911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.624891    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3d132e-8479-4c24-a5a1-8dcc1d404880-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.625192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.625175    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3d132e-8479-4c24-a5a1-8dcc1d404880-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.680097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.680060    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt7x\" (UniqueName: \"kubernetes.io/projected/cc3d132e-8479-4c24-a5a1-8dcc1d404880-kube-api-access-2tt7x\") pod \"echo-simulator-kserve-64b6f8f96c-866l2\" (UID: \"cc3d132e-8479-4c24-a5a1-8dcc1d404880\") " pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:44.782991 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:44.782950    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:45.153071 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:45.153041    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-866l2"]
Nov 01 08:42:45.155569 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:42:45.155527    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3d132e_8479_4c24_a5a1_8dcc1d404880.slice/crio-3e62ebae004999d1efef8c5c6e5e63345afa9aa04f620eed0e0221edfac2ddab WatchSource:0}: Error finding container 3e62ebae004999d1efef8c5c6e5e63345afa9aa04f620eed0e0221edfac2ddab: Status 404 returned error can't find the container with id 3e62ebae004999d1efef8c5c6e5e63345afa9aa04f620eed0e0221edfac2ddab
Nov 01 08:42:45.574289 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:45.574253    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2" event={"ID":"cc3d132e-8479-4c24-a5a1-8dcc1d404880","Type":"ContainerStarted","Data":"3e62ebae004999d1efef8c5c6e5e63345afa9aa04f620eed0e0221edfac2ddab"}
Nov 01 08:42:47.460235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:47.460185    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:47.704033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:47.703989    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:47.720802 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:47.720720    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:48.591344 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:48.591306    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2" event={"ID":"cc3d132e-8479-4c24-a5a1-8dcc1d404880","Type":"ContainerStarted","Data":"3d6b029f5c0c7a82891b5e33938c2ad7368732dd2962bd276d679ade742bf072"}
Nov 01 08:42:48.602543 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:48.602507    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xcr5m"
Nov 01 08:42:48.702754 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:48.702704    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2" podStartSLOduration=2.049067935 podStartE2EDuration="4.702688968s" totalImagesPullingTime="2.653621033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:48.70163371 +0000 UTC m=+641.896028267" watchObservedRunningTime="2026-11-01 08:42:48.702688968 +0000 UTC m=+641.897083559"
Nov 01 08:42:48.783602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:48.783562    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:50.761682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.761646    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"]
Nov 01 08:42:50.765380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.765357    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.814210 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.814172    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 08:42:50.814210 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.814189    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\""
Nov 01 08:42:50.863171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.863141    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"]
Nov 01 08:42:50.878042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878004    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878201 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878058    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878201 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878148    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878275 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878245    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878269    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878399 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878381    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878446 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878429    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878484 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878458    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.878580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.878558    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rw5\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.956858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.956813    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:50.979021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.978986    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979023    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979047    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979064    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979183    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979254    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rw5\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979304    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979337    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979366    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979646 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979444    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979646 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979507    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.979830 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.979810    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.980032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.980009    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.980117 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.980098    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.981697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.981675    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:50.981959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:50.981940    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:51.043606 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.043519    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rw5\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:51.044103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.044081    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5n9td\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:51.077998 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.077964    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:51.102229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.102189    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:51.209440 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.209409    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"]
Nov 01 08:42:51.213254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.213232    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.261994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.261963    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 08:42:51.268000 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.267972    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"]
Nov 01 08:42:51.282023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.281986    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.282023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.282022    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a83d0f-f6cd-4498-9982-6dae24b6639b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.282220 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.282052    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.282220 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.282191    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.282286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.282229    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.282286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.282260    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjjr\" (UniqueName: \"kubernetes.io/projected/52a83d0f-f6cd-4498-9982-6dae24b6639b-kube-api-access-zxjjr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.302183 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.302102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.383711    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.383770    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjjr\" (UniqueName: \"kubernetes.io/projected/52a83d0f-f6cd-4498-9982-6dae24b6639b-kube-api-access-zxjjr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.383861    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.383892    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a83d0f-f6cd-4498-9982-6dae24b6639b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.383948    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.384014    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.384498    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.384796    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.385701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.385367    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.389166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.389017    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a83d0f-f6cd-4498-9982-6dae24b6639b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.391295 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.391272    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a83d0f-f6cd-4498-9982-6dae24b6639b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.442135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.442098    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjjr\" (UniqueName: \"kubernetes.io/projected/52a83d0f-f6cd-4498-9982-6dae24b6639b-kube-api-access-zxjjr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np\" (UID: \"52a83d0f-f6cd-4498-9982-6dae24b6639b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.489970 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.489944    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"]
Nov 01 08:42:51.491686 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:42:51.491663    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7fcfe5_6a1d_4ff3_8634_7c151b155a5b.slice/crio-e04b3d7f8736a7890a948a868170c54ef906a7e370be57d5e8e8686f4962b506 WatchSource:0}: Error finding container e04b3d7f8736a7890a948a868170c54ef906a7e370be57d5e8e8686f4962b506: Status 404 returned error can't find the container with id e04b3d7f8736a7890a948a868170c54ef906a7e370be57d5e8e8686f4962b506
Nov 01 08:42:51.525483 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.525447    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:51.607508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.607469    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" event={"ID":"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b","Type":"ContainerStarted","Data":"e04b3d7f8736a7890a948a868170c54ef906a7e370be57d5e8e8686f4962b506"}
Nov 01 08:42:51.890009 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:51.889988    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"]
Nov 01 08:42:51.892155 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:42:51.892125    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52a83d0f_f6cd_4498_9982_6dae24b6639b.slice/crio-840f434585cc555c46e0819185d50b77d2d0bef330194cb372561aa558589934 WatchSource:0}: Error finding container 840f434585cc555c46e0819185d50b77d2d0bef330194cb372561aa558589934: Status 404 returned error can't find the container with id 840f434585cc555c46e0819185d50b77d2d0bef330194cb372561aa558589934
Nov 01 08:42:52.614426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:52.614384    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" event={"ID":"52a83d0f-f6cd-4498-9982-6dae24b6639b","Type":"ContainerStarted","Data":"829ac99c053ebace3adc1f7c79dc33a22bba2854080ca1cde3292c13fffef091"}
Nov 01 08:42:52.614426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:52.614433    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" event={"ID":"52a83d0f-f6cd-4498-9982-6dae24b6639b","Type":"ContainerStarted","Data":"840f434585cc555c46e0819185d50b77d2d0bef330194cb372561aa558589934"}
Nov 01 08:42:52.724125 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:52.724071    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podStartSLOduration=1.123203169 podStartE2EDuration="1.724056632s" totalImagesPullingTime="600.853463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:52.723588566 +0000 UTC m=+645.917983123" watchObservedRunningTime="2026-11-01 08:42:52.724056632 +0000 UTC m=+645.918451186"
Nov 01 08:42:53.526393 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:53.526362    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:42:54.784185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:54.784146    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:54.799487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:54.799458    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:55.646435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:55.646403    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-866l2"
Nov 01 08:42:56.456969 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:56.456934    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:42:56.457279 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:56.457005    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:42:56.457279 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:56.457058    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:42:56.636763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:56.636726    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" event={"ID":"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b","Type":"ContainerStarted","Data":"04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a"}
Nov 01 08:42:56.798190 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:56.798132    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podStartSLOduration=1.835698292 podStartE2EDuration="6.798117022s" totalImagesPullingTime="4.96241873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:56.795149396 +0000 UTC m=+649.989543941" watchObservedRunningTime="2026-11-01 08:42:56.798117022 +0000 UTC m=+649.992511576"
Nov 01 08:42:57.081827 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:57.081729    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:42:57.083148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:57.083109    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:42:57.083283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:57.083183    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:42:58.079556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:58.078959    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:42:58.079556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:58.079028    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:42:59.079558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:59.078936    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:42:59.079558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:42:59.079001    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:43:00.079558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:00.078960    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:43:00.079558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:00.079023    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:43:01.078778 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:01.078739    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:43:01.079106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:01.079074    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:43:01.526463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:01.526421    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:43:01.538750 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:01.538717    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:43:01.693588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:01.693519    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:43:02.079991 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:02.079953    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:43:02.080296 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:02.080267    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:43:03.079423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:03.079384    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:43:03.079954 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:03.079464    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:43:03.403701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:03.403595    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:04.078673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:04.078634    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5n9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:43:04.078837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:04.078691    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 08:43:05.853816 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:05.853781    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:43:05.854310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:05.854158    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:43:05.855054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:05.855033    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 08:43:05.900358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:05.900322    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"]
Nov 01 08:43:05.904073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:05.904055    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.030329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.030291    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"]
Nov 01 08:43:06.045900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.045868    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.045909    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.045948    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.046035    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.046080    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.046107    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.046290    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.046333    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd487\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.046421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.046382    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147361    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147407    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147454    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147470    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd487\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147488    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147516    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147572    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147627    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.147780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147657    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.148086 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.147876    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.148137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.148081    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.148137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.148102    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.148137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.148111    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.148426 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.148403    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.149925 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.149902    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.150047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.150029    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.204574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.204516    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.204705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.204645    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd487\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487\") pod \"maas-default-gateway-openshift-default-746ff8f49f-lnjmm\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.216425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.216395    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:06.642780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.642751    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"]
Nov 01 08:43:06.644933 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:43:06.644898    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119bbcf8_3604_44d5_918c_7b5c5f10aabd.slice/crio-315695bcd02ce537549b926b3dcf765d6aa1a2220fff9863567eb2019111501a WatchSource:0}: Error finding container 315695bcd02ce537549b926b3dcf765d6aa1a2220fff9863567eb2019111501a: Status 404 returned error can't find the container with id 315695bcd02ce537549b926b3dcf765d6aa1a2220fff9863567eb2019111501a
Nov 01 08:43:06.647951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.647916    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:06.648050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.647998    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:06.648112 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.648045    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:06.703524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:06.703488    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" event={"ID":"119bbcf8-3604-44d5-918c-7b5c5f10aabd","Type":"ContainerStarted","Data":"315695bcd02ce537549b926b3dcf765d6aa1a2220fff9863567eb2019111501a"}
Nov 01 08:43:07.710444 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:07.710408    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" event={"ID":"119bbcf8-3604-44d5-918c-7b5c5f10aabd","Type":"ContainerStarted","Data":"33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263"}
Nov 01 08:43:07.939221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:07.939142    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podStartSLOduration=2.939124238 podStartE2EDuration="2.939124238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:07.892773692 +0000 UTC m=+661.087168247" watchObservedRunningTime="2026-11-01 08:43:07.939124238 +0000 UTC m=+661.133518796"
Nov 01 08:43:08.217495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:08.217457    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:08.219040 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:08.219016    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:08.219109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:08.219069    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:09.217469 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:09.217433    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:09.217910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:09.217488    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:10.217571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:10.217514    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:10.217985 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:10.217592    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:11.216994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:11.216955    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:11.217188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:11.217021    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:11.324239 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:43:11.324185    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:43:12.217820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:12.217780    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:12.217996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:12.217846    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:13.217910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:13.217868    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:13.218305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:13.217924    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:14.217651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:14.217613    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 08:43:14.217858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:14.217674    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 08:43:15.884419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:15.884351    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:15.884912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:15.884797    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:15.885726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:15.885706    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 08:43:19.852333 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:19.852233    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:20.000367 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:20.000324    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:21.400759 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:21.400720    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:21.540942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:21.540909    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:41.321226 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:43:41.321189    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:43:48.869420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:48.869377    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:49.022610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.022576    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:49.177752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.177658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:49.389593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.389517    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:49.540625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.540582    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"]
Nov 01 08:43:49.549334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.549309    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.598342 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.598312    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-395735a2-openshift-default-dockercfg-9q5rq\""
Nov 01 08:43:49.643485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.643448    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:49.649030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.648988    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"]
Nov 01 08:43:49.667310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667267    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667475 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667318    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nsvm\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667517 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667463    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667511    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667638 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667619    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667648    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667664    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667812    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.667865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.667846    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769325    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769586 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769380    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769586 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769492    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769586 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769557    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769639    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769665    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769724    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769750    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nsvm\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769799    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.769905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.769810    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.770119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.770095    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.770256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.770185    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.770256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.770206    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.770368 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.770352    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.771821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.771800    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.772183 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.772164    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.799916 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.799821    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:49.827905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.827878    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.828127 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.828101    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nsvm\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm\") pod \"e2e-ait-395735a2-openshift-default-548dddf77f-zc582\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") " pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.862780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.862747    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:49.955694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:49.955658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:50.311366 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.311330    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:50.321775 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.321746    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"]
Nov 01 08:43:50.323419 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:43:50.323395    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0 WatchSource:0}: Error finding container 26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0: Status 404 returned error can't find the container with id 26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0
Nov 01 08:43:50.325665 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.325625    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:50.325725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.325705    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:50.325761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.325738    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:50.463122 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.463086    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"]
Nov 01 08:43:50.466691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.466670    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.515654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.515625    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f814ac10-gw-openshift-default-dockercfg-dhnnz\""
Nov 01 08:43:50.525435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.525406    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"]
Nov 01 08:43:50.556372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.556332    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:50.578447 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578354    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578447 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578426    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578610    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578675    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578726    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578881 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578778    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578888    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578917    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.578952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.578942    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqjf\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.679740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.679702    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.679939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.679774    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.679939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.679856    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.679939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.679884    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.679939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.679918    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqjf\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.679956    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.680022    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.680054    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.680080    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680366 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.680253    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680366 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.680278    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.680791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.680504    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.681039 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.681017    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.681644 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.681605    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.682700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.682674    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.683171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.683152    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.737796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.737761    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.737942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.737867    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqjf\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf\") pod \"e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") " pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.765830 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.765793    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:50.779883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.779851    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:50.914656 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.914613    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:50.917244 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.917216    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" event={"ID":"a6711ca6-c9f6-4178-b6f3-b206262f4b0c","Type":"ContainerStarted","Data":"3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"}
Nov 01 08:43:50.917361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:50.917273    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" event={"ID":"a6711ca6-c9f6-4178-b6f3-b206262f4b0c","Type":"ContainerStarted","Data":"26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0"}
Nov 01 08:43:51.036952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.036681    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podStartSLOduration=2.036660137 podStartE2EDuration="2.036660137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:51.035853974 +0000 UTC m=+704.230248547" watchObservedRunningTime="2026-11-01 08:43:51.036660137 +0000 UTC m=+704.231054694"
Nov 01 08:43:51.196783 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.196758    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"]
Nov 01 08:43:51.198600 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:43:51.198564    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice/crio-c6776587a9c961bb377f1850108c36995a378e0fe49763f1e2f9c73020bbda9c WatchSource:0}: Error finding container c6776587a9c961bb377f1850108c36995a378e0fe49763f1e2f9c73020bbda9c: Status 404 returned error can't find the container with id c6776587a9c961bb377f1850108c36995a378e0fe49763f1e2f9c73020bbda9c
Nov 01 08:43:51.200852 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.200816    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:51.200934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.200908    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:51.200977 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.200959    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:51.863244 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.863205    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:51.868521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.868492    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:51.924233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.924196    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" event={"ID":"4eaabd03-3f6c-40ce-82f6-f73902fb6591","Type":"ContainerStarted","Data":"36205918b4bbc0dce3f94e1acfb1e73b529a62398dd88eac498372e79cb03178"}
Nov 01 08:43:51.924433 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.924258    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" event={"ID":"4eaabd03-3f6c-40ce-82f6-f73902fb6591","Type":"ContainerStarted","Data":"c6776587a9c961bb377f1850108c36995a378e0fe49763f1e2f9c73020bbda9c"}
Nov 01 08:43:51.924597 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.924571    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:51.925507 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.925488    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:43:51.966590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:51.966528    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:52.187577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:52.187478    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:52.208221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:52.208158    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podStartSLOduration=2.20814223 podStartE2EDuration="2.20814223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:52.207887767 +0000 UTC m=+705.402282323" watchObservedRunningTime="2026-11-01 08:43:52.20814223 +0000 UTC m=+705.402536853"
Nov 01 08:43:52.780964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:52.780922    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:52.786186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:52.786157    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:52.930090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:52.930058    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:52.930944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:52.930920    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:43:53.208962 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:53.208877    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:53.369569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:53.369522    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:53.518195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:53.518162    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:54.008164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:54.008132    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:54.213553 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:54.213512    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:54.452105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:54.452027    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:55.822525 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:55.822490    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:55.973049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:55.973015    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:56.159580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.159469    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:56.354321 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.354277    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:56.528249 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.528213    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:56.667021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.666983    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"]
Nov 01 08:43:56.670513 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.670487    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.719078 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.719043    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0f765cd6-openshift-default-dockercfg-4fckl\""
Nov 01 08:43:56.764276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.764233    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:56.767644 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.767617    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"]
Nov 01 08:43:56.870157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870063    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.870722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.870905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870777    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.870905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870810    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.870905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870876    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.871322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870936    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prscb\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.871322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.870973    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.871322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.871008    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.871322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.871039    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972114    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972151    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972295    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972333    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972394    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972453    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prscb\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972585 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972487    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972585 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972525    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972584    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972623    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972713    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.972847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972790    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.973004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.972980    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.973578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.973518    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.975316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.975289    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:56.975587 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:56.975567    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:57.029690 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.029645    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:57.029867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.029769    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prscb\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:57.071455 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.071421    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:57.283777 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.283740    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:57.698784 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.698711    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"]
Nov 01 08:43:57.702018 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:43:57.701991    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f7e959_1747_46a6_8b9e_66191451ee20.slice/crio-a79bce1865b221e1d85e19e24466b3ffaa0fdfb1b3039898093a4c814647a126 WatchSource:0}: Error finding container a79bce1865b221e1d85e19e24466b3ffaa0fdfb1b3039898093a4c814647a126: Status 404 returned error can't find the container with id a79bce1865b221e1d85e19e24466b3ffaa0fdfb1b3039898093a4c814647a126
Nov 01 08:43:57.707264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.707227    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:57.707357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.707303    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:57.707398 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.707332    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:43:57.906676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.906638    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:57.955795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.955708    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" event={"ID":"b3f7e959-1747-46a6-8b9e-66191451ee20","Type":"ContainerStarted","Data":"6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"}
Nov 01 08:43:57.955795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:57.955762    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" event={"ID":"b3f7e959-1747-46a6-8b9e-66191451ee20","Type":"ContainerStarted","Data":"a79bce1865b221e1d85e19e24466b3ffaa0fdfb1b3039898093a4c814647a126"}
Nov 01 08:43:58.070752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.070695    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podStartSLOduration=2.070679472 podStartE2EDuration="2.070679472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:58.068231575 +0000 UTC m=+711.262626132" watchObservedRunningTime="2026-11-01 08:43:58.070679472 +0000 UTC m=+711.265074027"
Nov 01 08:43:58.110973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.110939    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:58.261920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.261883    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:58.284054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.284016    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:58.306840 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.306809    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:58.549429 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.549350    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:58.811378 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.811297    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:43:58.960550 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.960499    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:58.961452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:58.961427    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:43:59.916701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:43:59.916654    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:00.118760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:00.118706    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:00.279867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:00.279832    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:00.884406 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:00.884374    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:01.044910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.044878    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:01.284485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.284442    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"]
Nov 01 08:44:01.290707 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.289301    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.317418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.317378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.317621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.317513    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vjn\" (UniqueName: \"kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn\") pod \"payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.343658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.343617    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"]
Nov 01 08:44:01.386286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.386241    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-685688c3\""
Nov 01 08:44:01.386493 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.386359    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-685688c3-dockercfg-gqhlr\""
Nov 01 08:44:01.418983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.418947    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.419172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.419015    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vjn\" (UniqueName: \"kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn\") pod \"payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.419722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.419697    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.434436 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.434398    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"]
Nov 01 08:44:01.437963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.437942    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.477070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.477033    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vjn\" (UniqueName: \"kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn\") pod \"payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.492614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.492575    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"]
Nov 01 08:44:01.527557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.520328    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume\") pod \"payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") " pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.527557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.520376    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pzr\" (UniqueName: \"kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr\") pod \"payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") " pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.603841 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.603760    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:01.621208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.621176    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume\") pod \"payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") " pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.621361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.621214    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pzr\" (UniqueName: \"kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr\") pod \"payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") " pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.622178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.622151    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume\") pod \"payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") " pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.679633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.679602    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pzr\" (UniqueName: \"kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr\") pod \"payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") " pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.751454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.751416    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:01.970559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:01.970521    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"]
Nov 01 08:44:01.972306 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:01.972274    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800340d0_8bfd_4159_9552_a63d9bbdf887.slice/crio-dc762daa3be90a6256862c94f1d8d91945f2c772332fad8d57487652d62aba81 WatchSource:0}: Error finding container dc762daa3be90a6256862c94f1d8d91945f2c772332fad8d57487652d62aba81: Status 404 returned error can't find the container with id dc762daa3be90a6256862c94f1d8d91945f2c772332fad8d57487652d62aba81
Nov 01 08:44:02.120628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:02.120603    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"]
Nov 01 08:44:02.122665 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:02.122634    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d17b28_1b1c_4117_b77c_a4f00b2c8786.slice/crio-5781d82765958c338fe890560633307b667e96e92743671dc830aaabd5a6d2be WatchSource:0}: Error finding container 5781d82765958c338fe890560633307b667e96e92743671dc830aaabd5a6d2be: Status 404 returned error can't find the container with id 5781d82765958c338fe890560633307b667e96e92743671dc830aaabd5a6d2be
Nov 01 08:44:02.982589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:02.982553    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c" event={"ID":"05d17b28-1b1c-4117-b77c-a4f00b2c8786","Type":"ContainerStarted","Data":"5781d82765958c338fe890560633307b667e96e92743671dc830aaabd5a6d2be"}
Nov 01 08:44:02.983773 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:02.983748    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8" event={"ID":"800340d0-8bfd-4159-9552-a63d9bbdf887","Type":"ContainerStarted","Data":"dc762daa3be90a6256862c94f1d8d91945f2c772332fad8d57487652d62aba81"}
Nov 01 08:44:03.244592 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.244483    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:03.408200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.408159    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:03.635821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.635736    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:03.643573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.643542    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"]
Nov 01 08:44:03.647657 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.647631    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.737491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.737455    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"]
Nov 01 08:44:03.744503 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.744448    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f814ac10\""
Nov 01 08:44:03.745006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.744977    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fsp\" (UniqueName: \"kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp\") pod \"payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.745180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.745157    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.745180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.745172    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f814ac10-dockercfg-trsw7\""
Nov 01 08:44:03.785940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.785838    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:03.846028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.845978    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.846236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.846154    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fsp\" (UniqueName: \"kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp\") pod \"payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.846982 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.846953    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.903926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.903836    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fsp\" (UniqueName: \"kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp\") pod \"payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:03.945801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.945727    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:03.965839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:03.965787    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:04.347965 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:04.347861    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"]
Nov 01 08:44:04.349194 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:04.349163    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99 WatchSource:0}: Error finding container 3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99: Status 404 returned error can't find the container with id 3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99
Nov 01 08:44:05.003052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:05.003017    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm" event={"ID":"fda4d0b4-d80c-44b6-b078-f8360ace57a7","Type":"ContainerStarted","Data":"3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99"}
Nov 01 08:44:05.122423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:05.122376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:05.326820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:05.326785    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:05.499794 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:05.499754    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:06.009620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.009574    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c" event={"ID":"05d17b28-1b1c-4117-b77c-a4f00b2c8786","Type":"ContainerStarted","Data":"4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb"}
Nov 01 08:44:06.009815 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.009774    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:06.011183 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.011159    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8" event={"ID":"800340d0-8bfd-4159-9552-a63d9bbdf887","Type":"ContainerStarted","Data":"b4ac7e361fc4d94512b73a606fea8160d70600b5f5a974099cda7e903205ee7a"}
Nov 01 08:44:06.011285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.011208    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:06.012628 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.012603    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm" event={"ID":"fda4d0b4-d80c-44b6-b078-f8360ace57a7","Type":"ContainerStarted","Data":"ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322"}
Nov 01 08:44:06.012735 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.012723    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:06.135109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.135049    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c" podStartSLOduration=1.70931888 podStartE2EDuration="5.135033394s" totalImagesPullingTime="3.425714514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:06.133936431 +0000 UTC m=+719.328330988" watchObservedRunningTime="2026-11-01 08:44:06.135033394 +0000 UTC m=+719.329427950"
Nov 01 08:44:06.251097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.251013    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8" podStartSLOduration=1.9453400950000002 podStartE2EDuration="5.25099021s" totalImagesPullingTime="3.305650115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:06.247134844 +0000 UTC m=+719.441529400" watchObservedRunningTime="2026-11-01 08:44:06.25099021 +0000 UTC m=+719.445384765"
Nov 01 08:44:06.357431 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:06.357309    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm" podStartSLOduration=2.428855151 podStartE2EDuration="3.357288938s" totalImagesPullingTime="928.433787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:06.354493688 +0000 UTC m=+719.548888243" watchObservedRunningTime="2026-11-01 08:44:06.357288938 +0000 UTC m=+719.551683493"
Nov 01 08:44:07.769156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:07.769125    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:07.936186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:07.936151    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:08.174176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.174099    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"]
Nov 01 08:44:08.178924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.178908    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.233648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.232473    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"]
Nov 01 08:44:08.275800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.275733    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5c25ae7a\""
Nov 01 08:44:08.275986 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.275953    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5c25ae7a-dockercfg-kkbzf\""
Nov 01 08:44:08.294519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.294492    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952pm\" (UniqueName: \"kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm\") pod \"payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.294680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.294561    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.396089 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.396049    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952pm\" (UniqueName: \"kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm\") pod \"payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.396279 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.396108    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.396813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.396795    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.464658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.464591    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952pm\" (UniqueName: \"kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm\") pod \"payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.492245 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.492212    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:08.855219 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:08.855194    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"]
Nov 01 08:44:08.857365 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:08.857340    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659e39ad_f338_413b_be07_2e54eba7ee55.slice/crio-19c1a4b0f51446e35118a76517d6d0e24b3c450809d05b93a09f12e19917eabf WatchSource:0}: Error finding container 19c1a4b0f51446e35118a76517d6d0e24b3c450809d05b93a09f12e19917eabf: Status 404 returned error can't find the container with id 19c1a4b0f51446e35118a76517d6d0e24b3c450809d05b93a09f12e19917eabf
Nov 01 08:44:09.029250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:09.029213    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx" event={"ID":"659e39ad-f338-413b-be07-2e54eba7ee55","Type":"ContainerStarted","Data":"d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc"}
Nov 01 08:44:09.029438 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:09.029420    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:09.029520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:09.029460    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx" event={"ID":"659e39ad-f338-413b-be07-2e54eba7ee55","Type":"ContainerStarted","Data":"19c1a4b0f51446e35118a76517d6d0e24b3c450809d05b93a09f12e19917eabf"}
Nov 01 08:44:09.138424 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:09.138318    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx" podStartSLOduration=1.138304415 podStartE2EDuration="1.138304415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:09.136411456 +0000 UTC m=+722.330806012" watchObservedRunningTime="2026-11-01 08:44:09.138304415 +0000 UTC m=+722.332698970"
Nov 01 08:44:10.440185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:10.440144    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:10.614704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:10.614672    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:10.759182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:10.759152    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:11.320461 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:44:11.320424    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:44:14.502119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.502085    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:14.664550 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.664497    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:14.907320 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.907222    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"]
Nov 01 08:44:14.911058 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.911042    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:14.961870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.961844    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0f765cd6-dockercfg-p45lw\""
Nov 01 08:44:14.961995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.961939    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0f765cd6\""
Nov 01 08:44:14.965423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:14.965396    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"]
Nov 01 08:44:15.058317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.058281    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.058662 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.058641    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw\") pod \"payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.067504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.067479    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"]
Nov 01 08:44:15.071127 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.071112    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.124816 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.124786    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"]
Nov 01 08:44:15.159981 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.159904    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.159981 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.159948    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5nkg\" (UniqueName: \"kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg\") pod \"payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") " pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.160207 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.160006    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw\") pod \"payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.160207 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.160029    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume\") pod \"payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") " pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.160647 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.160622    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.216993 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.216965    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw\") pod \"payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.223844 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.223802    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:15.260974 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.260937    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume\") pod \"payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") " pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.261104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.261057    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5nkg\" (UniqueName: \"kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg\") pod \"payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") " pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.261735 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.261711    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume\") pod \"payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") " pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.318400 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.318366    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5nkg\" (UniqueName: \"kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg\") pod \"payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") " pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.384810 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.384783    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:15.588224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.588199    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"]
Nov 01 08:44:15.590315 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:15.590284    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-f89367adc57173da03a128dec3bbb59293fc1b5d489a24885524faf3f1289877 WatchSource:0}: Error finding container f89367adc57173da03a128dec3bbb59293fc1b5d489a24885524faf3f1289877: Status 404 returned error can't find the container with id f89367adc57173da03a128dec3bbb59293fc1b5d489a24885524faf3f1289877
Nov 01 08:44:15.624654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.624627    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:15.751083 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.751058    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"]
Nov 01 08:44:15.752190 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:15.752156    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebf7f2_282f_41c5_bd4a_99b995980dfc.slice/crio-da0e69af24e83d159ee24d954693ce383e1a1b97969917ec831b1f07bc5ef248 WatchSource:0}: Error finding container da0e69af24e83d159ee24d954693ce383e1a1b97969917ec831b1f07bc5ef248: Status 404 returned error can't find the container with id da0e69af24e83d159ee24d954693ce383e1a1b97969917ec831b1f07bc5ef248
Nov 01 08:44:15.823471 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.823438    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:15.971557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:15.971502    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:16.064633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.064558    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr" event={"ID":"28ebf7f2-282f-41c5-bd4a-99b995980dfc","Type":"ContainerStarted","Data":"9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d"}
Nov 01 08:44:16.064633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.064614    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr" event={"ID":"28ebf7f2-282f-41c5-bd4a-99b995980dfc","Type":"ContainerStarted","Data":"da0e69af24e83d159ee24d954693ce383e1a1b97969917ec831b1f07bc5ef248"}
Nov 01 08:44:16.064845 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.064731    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:16.066136 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.066114    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns" event={"ID":"10c6301d-730f-4367-85d9-abd75b92bdea","Type":"ContainerStarted","Data":"4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e"}
Nov 01 08:44:16.066260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.066146    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns" event={"ID":"10c6301d-730f-4367-85d9-abd75b92bdea","Type":"ContainerStarted","Data":"f89367adc57173da03a128dec3bbb59293fc1b5d489a24885524faf3f1289877"}
Nov 01 08:44:16.066260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.066180    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:16.175726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.175672    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr" podStartSLOduration=1.175658741 podStartE2EDuration="1.175658741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:16.174767119 +0000 UTC m=+729.369161673" watchObservedRunningTime="2026-11-01 08:44:16.175658741 +0000 UTC m=+729.370053290"
Nov 01 08:44:16.286738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:16.286673    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns" podStartSLOduration=2.286655229 podStartE2EDuration="2.286655229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:16.286117836 +0000 UTC m=+729.480512417" watchObservedRunningTime="2026-11-01 08:44:16.286655229 +0000 UTC m=+729.481049786"
Nov 01 08:44:17.020227 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:17.020201    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:17.020616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:17.020266    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:17.020616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:17.020401    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:44:20.037472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:20.037241    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:44:20.685095 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:20.685027    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:22.965746 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:22.965704    4314 patch_prober.go:29] interesting pod/e2e-ait-395735a2-openshift-default-548dddf77f-zc582 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:44:22.966164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:22.965770    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:44:25.529651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:25.529609    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:25.694934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:25.694898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:25.841620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:25.841528    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:26.135903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:26.135821    4314 generic.go:356] "Generic (PLEG): container finished" podID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerID="3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210" exitCode=0
Nov 01 08:44:26.135903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:26.135870    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" event={"ID":"a6711ca6-c9f6-4178-b6f3-b206262f4b0c","Type":"ContainerDied","Data":"3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"}
Nov 01 08:44:26.136240 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:26.136225    4314 scope.go:122] "RemoveContainer" containerID="3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"
Nov 01 08:44:26.137526 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:26.137494    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:26.137629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:26.137594    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:26.137668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:26.137631    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:27.107766 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:27.107734    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:44:27.110982 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:27.110960    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:44:27.142224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:27.142192    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" event={"ID":"a6711ca6-c9f6-4178-b6f3-b206262f4b0c","Type":"ContainerStarted","Data":"b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889"}
Nov 01 08:44:27.869556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:27.865961    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:44:27.869556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:27.866131    4314 patch_prober.go:29] interesting pod/e2e-ait-395735a2-openshift-default-548dddf77f-zc582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 08:44:27.869556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:27.866178    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 08:44:28.866017 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:28.865979    4314 patch_prober.go:29] interesting pod/e2e-ait-395735a2-openshift-default-548dddf77f-zc582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 08:44:28.866647 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:28.866612    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 08:44:29.881001 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:29.880965    4314 patch_prober.go:29] interesting pod/e2e-ait-395735a2-openshift-default-548dddf77f-zc582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 08:44:29.881604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:29.881574    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 08:44:30.498425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:30.498385    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:30.867495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:30.866972    4314 patch_prober.go:29] interesting pod/e2e-ait-395735a2-openshift-default-548dddf77f-zc582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 08:44:30.867495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:30.867042    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 08:44:31.875064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:31.875028    4314 patch_prober.go:29] interesting pod/e2e-ait-395735a2-openshift-default-548dddf77f-zc582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 08:44:31.875710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:31.875672    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 08:44:32.867517 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:32.867478    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:44:32.867879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:32.867857    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:44:32.868684 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:32.868656    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:44:34.196938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:34.196903    4314 generic.go:356] "Generic (PLEG): container finished" podID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerID="36205918b4bbc0dce3f94e1acfb1e73b529a62398dd88eac498372e79cb03178" exitCode=0
Nov 01 08:44:34.197391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:34.196974    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" event={"ID":"4eaabd03-3f6c-40ce-82f6-f73902fb6591","Type":"ContainerDied","Data":"36205918b4bbc0dce3f94e1acfb1e73b529a62398dd88eac498372e79cb03178"}
Nov 01 08:44:34.197515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:34.197494    4314 scope.go:122] "RemoveContainer" containerID="36205918b4bbc0dce3f94e1acfb1e73b529a62398dd88eac498372e79cb03178"
Nov 01 08:44:34.198739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:34.198712    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:34.198798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:34.198785    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:34.198833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:34.198820    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:35.208742 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:35.208704    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" event={"ID":"4eaabd03-3f6c-40ce-82f6-f73902fb6591","Type":"ContainerStarted","Data":"9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837"}
Nov 01 08:44:35.394109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:35.394067    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:35.780967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:35.780930    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:44:35.782894 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:35.782864    4314 patch_prober.go:29] interesting pod/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 08:44:35.783107 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:35.783080    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 08:44:36.786557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:36.786097    4314 patch_prober.go:29] interesting pod/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 08:44:36.786557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:36.786176    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 08:44:37.781556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:37.781018    4314 patch_prober.go:29] interesting pod/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 08:44:37.781556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:37.781098    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 08:44:37.930625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:37.930591    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:44:38.782696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:38.782651    4314 patch_prober.go:29] interesting pod/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 08:44:38.782874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:38.782756    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 08:44:39.746830 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.746794    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"]
Nov 01 08:44:39.749629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.749605    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.783635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.783043    4314 patch_prober.go:29] interesting pod/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 08:44:39.783635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.783104    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 08:44:39.801349 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.801318    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:44:39.844225 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.844193    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:44:39.844397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.844237    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:44:39.844498 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.844435    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:44:39.847646 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.847598    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"]
Nov 01 08:44:39.896476 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.896429    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.896734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.896628    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwwrv\" (UniqueName: \"kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.896734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.896680    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.998047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.997951    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.998213 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.998068    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwwrv\" (UniqueName: \"kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.998213 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.998097    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:39.998744 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:39.998725    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:40.001108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.001089    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:40.056579 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.056545    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwwrv\" (UniqueName: \"kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv\") pod \"maas-api-revoke-keys-e2e-ait-685688c3-5s8zj\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:40.087190 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.086891    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:40.456260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.456176    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"]
Nov 01 08:44:40.461265 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:40.461222    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb573fc43_28ce_4a17_901c_aa732f2a2d0c.slice/crio-ff64a9e6109313a30b0d8391f4e5a36ecb5117d7dc399bc323e88cdcf2e628c4 WatchSource:0}: Error finding container ff64a9e6109313a30b0d8391f4e5a36ecb5117d7dc399bc323e88cdcf2e628c4: Status 404 returned error can't find the container with id ff64a9e6109313a30b0d8391f4e5a36ecb5117d7dc399bc323e88cdcf2e628c4
Nov 01 08:44:40.748048 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.748018    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:40.789303 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.789242    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:44:40.790938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:40.790914    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:44:41.242477 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:41.242443    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj" event={"ID":"b573fc43-28ce-4a17-901c-aa732f2a2d0c","Type":"ContainerStarted","Data":"ff64a9e6109313a30b0d8391f4e5a36ecb5117d7dc399bc323e88cdcf2e628c4"}
Nov 01 08:44:41.323897 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:44:41.323850    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:44:42.249907 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:42.249875    4314 generic.go:356] "Generic (PLEG): container finished" podID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerID="6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9" exitCode=0
Nov 01 08:44:42.250276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:42.249944    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" event={"ID":"b3f7e959-1747-46a6-8b9e-66191451ee20","Type":"ContainerDied","Data":"6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"}
Nov 01 08:44:42.250371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:42.250354    4314 scope.go:122] "RemoveContainer" containerID="6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"
Nov 01 08:44:42.251908 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:42.251874    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:42.251983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:42.251970    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:42.252033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:42.252017    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:44:43.287168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:43.287132    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" event={"ID":"b3f7e959-1747-46a6-8b9e-66191451ee20","Type":"ContainerStarted","Data":"fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593"}
Nov 01 08:44:43.983007 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:43.982977    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:44:44.284582 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:44.284483    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:44:44.285942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:44.285914    4314 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 08:44:44.286065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:44.285965    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 08:44:45.289286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:45.289239    4314 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 08:44:45.289786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:45.289300    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 08:44:46.027126 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.027096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:46.284872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.284834    4314 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 08:44:46.285103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.284898    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 08:44:46.307502 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.307470    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj" event={"ID":"b573fc43-28ce-4a17-901c-aa732f2a2d0c","Type":"ContainerStarted","Data":"4bf61e74d5bf6f3020f473fa658a01bed225b1ce85f1cb7b7e24d0f0ec91f99e"}
Nov 01 08:44:46.416764 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.416703    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj" podStartSLOduration=2.243207068 podStartE2EDuration="7.416688435s" totalImagesPullingTime="5.173481367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:46.414855885 +0000 UTC m=+759.609250474" watchObservedRunningTime="2026-11-01 08:44:46.416688435 +0000 UTC m=+759.611083009"
Nov 01 08:44:46.882641 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.882595    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"]
Nov 01 08:44:46.886926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.886906    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:46.937017 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.936992    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"]
Nov 01 08:44:46.989112 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.989077    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nb24\" (UniqueName: \"kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:46.989311 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.989135    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:46.989311 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:46.989198    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.090219 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.090187    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nb24\" (UniqueName: \"kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.090371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.090233    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.090371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.090342    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.090883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.090859    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.092636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.092615    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.146964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.146933    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nb24\" (UniqueName: \"kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24\") pod \"maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.199677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.199648    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:47.287570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.285918    4314 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 08:44:47.287570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.285980    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 08:44:47.316111 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.315672    4314 generic.go:356] "Generic (PLEG): container finished" podID="b573fc43-28ce-4a17-901c-aa732f2a2d0c" containerID="4bf61e74d5bf6f3020f473fa658a01bed225b1ce85f1cb7b7e24d0f0ec91f99e" exitCode=0
Nov 01 08:44:47.316111 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.315752    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj" event={"ID":"b573fc43-28ce-4a17-901c-aa732f2a2d0c","Type":"ContainerDied","Data":"4bf61e74d5bf6f3020f473fa658a01bed225b1ce85f1cb7b7e24d0f0ec91f99e"}
Nov 01 08:44:47.584432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:47.584327    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"]
Nov 01 08:44:47.589314 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:47.589288    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003182a4_0623_46c4_9f7c_774cfe50ab93.slice/crio-a2763ba5a4905cfb5047c75557381ca39e84084bd936ef443f0a19cf2d86308c WatchSource:0}: Error finding container a2763ba5a4905cfb5047c75557381ca39e84084bd936ef443f0a19cf2d86308c: Status 404 returned error can't find the container with id a2763ba5a4905cfb5047c75557381ca39e84084bd936ef443f0a19cf2d86308c
Nov 01 08:44:48.285202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.285168    4314 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 08:44:48.285372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.285223    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 08:44:48.322222 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.322184    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn" event={"ID":"003182a4-0623-46c4-9f7c-774cfe50ab93","Type":"ContainerStarted","Data":"511053eadd95800309d3ff1118d70e9bcbbc4b7cae56450f06cb35e4d1aed4c0"}
Nov 01 08:44:48.322688 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.322237    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn" event={"ID":"003182a4-0623-46c4-9f7c-774cfe50ab93","Type":"ContainerStarted","Data":"a2763ba5a4905cfb5047c75557381ca39e84084bd936ef443f0a19cf2d86308c"}
Nov 01 08:44:48.433056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.432996    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn" podStartSLOduration=2.432973695 podStartE2EDuration="2.432973695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:48.430637948 +0000 UTC m=+761.625032503" watchObservedRunningTime="2026-11-01 08:44:48.432973695 +0000 UTC m=+761.627368252"
Nov 01 08:44:48.457409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.457379    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:48.608353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.608263    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv\" (UniqueName: \"kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv\") pod \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") "
Nov 01 08:44:48.608353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.608303    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca\") pod \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") "
Nov 01 08:44:48.608622 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.608360    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca\") pod \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\" (UID: \"b573fc43-28ce-4a17-901c-aa732f2a2d0c\") "
Nov 01 08:44:48.608805 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.608777    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca" pod "b573fc43-28ce-4a17-901c-aa732f2a2d0c" (UID: "b573fc43-28ce-4a17-901c-aa732f2a2d0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:44:48.610616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.610593    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv" pod "b573fc43-28ce-4a17-901c-aa732f2a2d0c" (UID: "b573fc43-28ce-4a17-901c-aa732f2a2d0c"). InnerVolumeSpecName "kube-api-access-dwwrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:44:48.610683 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.610626    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca" pod "b573fc43-28ce-4a17-901c-aa732f2a2d0c" (UID: "b573fc43-28ce-4a17-901c-aa732f2a2d0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:44:48.710171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.710099    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:48.710171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.710129    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwwrv\" (UniqueName: \"kubernetes.io/projected/b573fc43-28ce-4a17-901c-aa732f2a2d0c-kube-api-access-dwwrv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:48.710171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:48.710140    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b573fc43-28ce-4a17-901c-aa732f2a2d0c-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:49.288880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.288852    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:44:49.289820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.289799    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:44:49.329204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.329172    4314 generic.go:356] "Generic (PLEG): container finished" podID="003182a4-0623-46c4-9f7c-774cfe50ab93" containerID="511053eadd95800309d3ff1118d70e9bcbbc4b7cae56450f06cb35e4d1aed4c0" exitCode=0
Nov 01 08:44:49.329593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.329220    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn" event={"ID":"003182a4-0623-46c4-9f7c-774cfe50ab93","Type":"ContainerDied","Data":"511053eadd95800309d3ff1118d70e9bcbbc4b7cae56450f06cb35e4d1aed4c0"}
Nov 01 08:44:49.330617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.330598    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"
Nov 01 08:44:49.330725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.330642    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj" event={"ID":"b573fc43-28ce-4a17-901c-aa732f2a2d0c","Type":"ContainerDied","Data":"ff64a9e6109313a30b0d8391f4e5a36ecb5117d7dc399bc323e88cdcf2e628c4"}
Nov 01 08:44:49.330725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:49.330676    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff64a9e6109313a30b0d8391f4e5a36ecb5117d7dc399bc323e88cdcf2e628c4"
Nov 01 08:44:50.397649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.397614    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"]
Nov 01 08:44:50.398338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.398309    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b573fc43-28ce-4a17-901c-aa732f2a2d0c" containerName="revoke-keys"
Nov 01 08:44:50.398338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.398333    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b573fc43-28ce-4a17-901c-aa732f2a2d0c" containerName="revoke-keys"
Nov 01 08:44:50.398479 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.398465    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b573fc43-28ce-4a17-901c-aa732f2a2d0c" containerName="revoke-keys"
Nov 01 08:44:50.402164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.402146    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.455484 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.455451    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"]
Nov 01 08:44:50.478407 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.478379    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:50.533427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.533389    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.533631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.533450    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.533631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.533579    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4cz\" (UniqueName: \"kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.634300 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634210    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca\") pod \"003182a4-0623-46c4-9f7c-774cfe50ab93\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") "
Nov 01 08:44:50.634473 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634329    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca\") pod \"003182a4-0623-46c4-9f7c-774cfe50ab93\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") "
Nov 01 08:44:50.634473 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634418    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24\" (UniqueName: \"kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24\") pod \"003182a4-0623-46c4-9f7c-774cfe50ab93\" (UID: \"003182a4-0623-46c4-9f7c-774cfe50ab93\") "
Nov 01 08:44:50.634648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634625    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca" pod "003182a4-0623-46c4-9f7c-774cfe50ab93" (UID: "003182a4-0623-46c4-9f7c-774cfe50ab93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:44:50.634695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634651    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.634762 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634744    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.634817 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634801    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4cz\" (UniqueName: \"kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.635231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.634923    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:50.635477 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.635443    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.636945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.636912    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca" pod "003182a4-0623-46c4-9f7c-774cfe50ab93" (UID: "003182a4-0623-46c4-9f7c-774cfe50ab93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:44:50.637057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.636959    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24" pod "003182a4-0623-46c4-9f7c-774cfe50ab93" (UID: "003182a4-0623-46c4-9f7c-774cfe50ab93"). InnerVolumeSpecName "kube-api-access-4nb24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:44:50.637453 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.637429    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.692303 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.692269    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4cz\" (UniqueName: \"kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz\") pod \"maas-api-revoke-keys-e2e-ait-395735a2-lmhd5\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.715952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.715924    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:50.735464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.735434    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003182a4-0623-46c4-9f7c-774cfe50ab93-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:50.735464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.735462    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nb24\" (UniqueName: \"kubernetes.io/projected/003182a4-0623-46c4-9f7c-774cfe50ab93-kube-api-access-4nb24\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:50.912301 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:50.912216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:51.084664 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:51.084628    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"]
Nov 01 08:44:51.342399 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:51.342361    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn" event={"ID":"003182a4-0623-46c4-9f7c-774cfe50ab93","Type":"ContainerDied","Data":"a2763ba5a4905cfb5047c75557381ca39e84084bd936ef443f0a19cf2d86308c"}
Nov 01 08:44:51.342399 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:51.342390    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"
Nov 01 08:44:51.342647 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:51.342422    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2763ba5a4905cfb5047c75557381ca39e84084bd936ef443f0a19cf2d86308c"
Nov 01 08:44:51.343798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:51.343767    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5" event={"ID":"ce5033b2-e7e4-4015-b797-649201e98482","Type":"ContainerStarted","Data":"d44e15b8b63992bb86120e998436c2ed1afb882948d0efd66d3bbe082d27c9e9"}
Nov 01 08:44:52.351436 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:52.351402    4314 generic.go:356] "Generic (PLEG): container finished" podID="ce5033b2-e7e4-4015-b797-649201e98482" containerID="10a9b7d45fdb451719e5f245c9a7d908685ef63a9ab8b9249c09ac4418f8fbdd" exitCode=0
Nov 01 08:44:52.351858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:52.351475    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5" event={"ID":"ce5033b2-e7e4-4015-b797-649201e98482","Type":"ContainerDied","Data":"10a9b7d45fdb451719e5f245c9a7d908685ef63a9ab8b9249c09ac4418f8fbdd"}
Nov 01 08:44:53.038182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.038141    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"]
Nov 01 08:44:53.138376 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.138338    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"]
Nov 01 08:44:53.379365 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.379330    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:53.502226 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.502203    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:53.519113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.519088    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:53.561203 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.561174    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca\") pod \"ce5033b2-e7e4-4015-b797-649201e98482\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") "
Nov 01 08:44:53.561397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.561271    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz\" (UniqueName: \"kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz\") pod \"ce5033b2-e7e4-4015-b797-649201e98482\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") "
Nov 01 08:44:53.561397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.561368    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca\") pod \"ce5033b2-e7e4-4015-b797-649201e98482\" (UID: \"ce5033b2-e7e4-4015-b797-649201e98482\") "
Nov 01 08:44:53.561665 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.561638    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca" pod "ce5033b2-e7e4-4015-b797-649201e98482" (UID: "ce5033b2-e7e4-4015-b797-649201e98482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:44:53.561805 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.561784    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:53.563612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.563582    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca" pod "ce5033b2-e7e4-4015-b797-649201e98482" (UID: "ce5033b2-e7e4-4015-b797-649201e98482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:44:53.563712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.563648    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz" pod "ce5033b2-e7e4-4015-b797-649201e98482" (UID: "ce5033b2-e7e4-4015-b797-649201e98482"). InnerVolumeSpecName "kube-api-access-7d4cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:44:53.662633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.662566    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d4cz\" (UniqueName: \"kubernetes.io/projected/ce5033b2-e7e4-4015-b797-649201e98482-kube-api-access-7d4cz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:53.662633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:53.662590    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5033b2-e7e4-4015-b797-649201e98482-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:54.362547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:54.362513    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"
Nov 01 08:44:54.362713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:54.362555    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5" event={"ID":"ce5033b2-e7e4-4015-b797-649201e98482","Type":"ContainerDied","Data":"d44e15b8b63992bb86120e998436c2ed1afb882948d0efd66d3bbe082d27c9e9"}
Nov 01 08:44:54.362713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:54.362597    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44e15b8b63992bb86120e998436c2ed1afb882948d0efd66d3bbe082d27c9e9"
Nov 01 08:44:55.566476 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.566440    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"]
Nov 01 08:44:55.567105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.567085    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ce5033b2-e7e4-4015-b797-649201e98482" containerName="revoke-keys"
Nov 01 08:44:55.567191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.567107    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5033b2-e7e4-4015-b797-649201e98482" containerName="revoke-keys"
Nov 01 08:44:55.567191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.567140    4314 cpu_manager.go:414] "containerMap: removing container" podUID="003182a4-0623-46c4-9f7c-774cfe50ab93" containerName="revoke-keys"
Nov 01 08:44:55.567191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.567149    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003182a4-0623-46c4-9f7c-774cfe50ab93" containerName="revoke-keys"
Nov 01 08:44:55.567404 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.567263    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="003182a4-0623-46c4-9f7c-774cfe50ab93" containerName="revoke-keys"
Nov 01 08:44:55.567404 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.567280    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5033b2-e7e4-4015-b797-649201e98482" containerName="revoke-keys"
Nov 01 08:44:55.570330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.570307    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.578994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.578967    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.579103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.579007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbsw\" (UniqueName: \"kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.579103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.579039    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.620685 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.620654    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:44:55.620840 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.620700    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:44:55.620840 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.620765    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:44:55.620967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.620952    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:44:55.623829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.623803    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"]
Nov 01 08:44:55.679613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.679584    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.679795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.679646    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbsw\" (UniqueName: \"kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.679795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.679705    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.680487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.680455    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.682117 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.682094    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.736774 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.736753    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbsw\" (UniqueName: \"kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw\") pod \"maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:55.884572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:55.884478    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:56.221979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:56.221892    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:56.250927 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:56.250899    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"]
Nov 01 08:44:56.252212 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:44:56.252177    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f8deed_2800_410b_8950_7b402fd936b2.slice/crio-57de3648097030954dd3eb061b91312a892c93f1d562a5ce94156fd80da36981 WatchSource:0}: Error finding container 57de3648097030954dd3eb061b91312a892c93f1d562a5ce94156fd80da36981: Status 404 returned error can't find the container with id 57de3648097030954dd3eb061b91312a892c93f1d562a5ce94156fd80da36981
Nov 01 08:44:56.373888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:56.373856    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx" event={"ID":"60f8deed-2800-410b-8950-7b402fd936b2","Type":"ContainerStarted","Data":"57de3648097030954dd3eb061b91312a892c93f1d562a5ce94156fd80da36981"}
Nov 01 08:44:57.381440 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:57.381355    4314 generic.go:356] "Generic (PLEG): container finished" podID="60f8deed-2800-410b-8950-7b402fd936b2" containerID="8d361d268ec8b2f864997f3a2b6276a6eecc4cce22d96fb0f49ad972a08f8833" exitCode=0
Nov 01 08:44:57.381857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:57.381448    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx" event={"ID":"60f8deed-2800-410b-8950-7b402fd936b2","Type":"ContainerDied","Data":"8d361d268ec8b2f864997f3a2b6276a6eecc4cce22d96fb0f49ad972a08f8833"}
Nov 01 08:44:57.788940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:57.788896    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"]
Nov 01 08:44:58.031466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.031428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:58.039062 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.038962    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c" podUID="05d17b28-1b1c-4117-b77c-a4f00b2c8786" containerName="payload-processing" containerID="cri-o://4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb" gracePeriod=25
Nov 01 08:44:58.139596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.139500    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8" podUID="800340d0-8bfd-4159-9552-a63d9bbdf887" containerName="payload-pre-processing" containerID="cri-o://b4ac7e361fc4d94512b73a606fea8160d70600b5f5a974099cda7e903205ee7a" gracePeriod=25
Nov 01 08:44:58.176027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.175989    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:44:58.306240 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.306212    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:58.389939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.389901    4314 generic.go:356] "Generic (PLEG): container finished" podID="05d17b28-1b1c-4117-b77c-a4f00b2c8786" containerID="4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb" exitCode=0
Nov 01 08:44:58.390480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.389963    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"
Nov 01 08:44:58.390480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.389969    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c" event={"ID":"05d17b28-1b1c-4117-b77c-a4f00b2c8786","Type":"ContainerDied","Data":"4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb"}
Nov 01 08:44:58.390480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.390194    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c" event={"ID":"05d17b28-1b1c-4117-b77c-a4f00b2c8786","Type":"ContainerDied","Data":"5781d82765958c338fe890560633307b667e96e92743671dc830aaabd5a6d2be"}
Nov 01 08:44:58.390480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.390220    4314 scope.go:122] "RemoveContainer" containerID="4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb"
Nov 01 08:44:58.391557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.391512    4314 generic.go:356] "Generic (PLEG): container finished" podID="800340d0-8bfd-4159-9552-a63d9bbdf887" containerID="b4ac7e361fc4d94512b73a606fea8160d70600b5f5a974099cda7e903205ee7a" exitCode=0
Nov 01 08:44:58.391780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.391762    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8" event={"ID":"800340d0-8bfd-4159-9552-a63d9bbdf887","Type":"ContainerDied","Data":"b4ac7e361fc4d94512b73a606fea8160d70600b5f5a974099cda7e903205ee7a"}
Nov 01 08:44:58.404610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.404590    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume\") pod \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") "
Nov 01 08:44:58.404694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.404678    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr\" (UniqueName: \"kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr\") pod \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\" (UID: \"05d17b28-1b1c-4117-b77c-a4f00b2c8786\") "
Nov 01 08:44:58.405022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.404995    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume" pod "05d17b28-1b1c-4117-b77c-a4f00b2c8786" (UID: "05d17b28-1b1c-4117-b77c-a4f00b2c8786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:44:58.406842 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.406821    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr" pod "05d17b28-1b1c-4117-b77c-a4f00b2c8786" (UID: "05d17b28-1b1c-4117-b77c-a4f00b2c8786"). InnerVolumeSpecName "kube-api-access-h4pzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:44:58.415456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.415430    4314 scope.go:122] "RemoveContainer" containerID="4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb"
Nov 01 08:44:58.416391 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:44:58.416356    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb\": container with ID starting with 4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb not found: ID does not exist" containerID="4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb"
Nov 01 08:44:58.416498 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.416401    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb"} err="failed to get container status \"4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb\": rpc error: code = NotFound desc = could not find container \"4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb\": container with ID starting with 4b70d7aa41e9aaa699b9fb06a61cf4e002a95e3431b3b8efc839bc12eb6326cb not found: ID does not exist"
Nov 01 08:44:58.420264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.420230    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:58.506176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.506145    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn\" (UniqueName: \"kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn\") pod \"800340d0-8bfd-4159-9552-a63d9bbdf887\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") "
Nov 01 08:44:58.506370 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.506203    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume\") pod \"800340d0-8bfd-4159-9552-a63d9bbdf887\" (UID: \"800340d0-8bfd-4159-9552-a63d9bbdf887\") "
Nov 01 08:44:58.506561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.506518    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume" pod "800340d0-8bfd-4159-9552-a63d9bbdf887" (UID: "800340d0-8bfd-4159-9552-a63d9bbdf887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:44:58.506561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.506549    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d17b28-1b1c-4117-b77c-a4f00b2c8786-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.506707 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.506570    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4pzr\" (UniqueName: \"kubernetes.io/projected/05d17b28-1b1c-4117-b77c-a4f00b2c8786-kube-api-access-h4pzr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.509620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.509574    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn" pod "800340d0-8bfd-4159-9552-a63d9bbdf887" (UID: "800340d0-8bfd-4159-9552-a63d9bbdf887"). InnerVolumeSpecName "kube-api-access-57vjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:44:58.530210 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.530188    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:58.607522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.607442    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw\" (UniqueName: \"kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw\") pod \"60f8deed-2800-410b-8950-7b402fd936b2\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") "
Nov 01 08:44:58.607695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.607559    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca\") pod \"60f8deed-2800-410b-8950-7b402fd936b2\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") "
Nov 01 08:44:58.607695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.607605    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca\") pod \"60f8deed-2800-410b-8950-7b402fd936b2\" (UID: \"60f8deed-2800-410b-8950-7b402fd936b2\") "
Nov 01 08:44:58.607901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.607869    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vjn\" (UniqueName: \"kubernetes.io/projected/800340d0-8bfd-4159-9552-a63d9bbdf887-kube-api-access-57vjn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.607901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.607897    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800340d0-8bfd-4159-9552-a63d9bbdf887-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.608052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.608022    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca" pod "60f8deed-2800-410b-8950-7b402fd936b2" (UID: "60f8deed-2800-410b-8950-7b402fd936b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:44:58.609529 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.609506    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw" pod "60f8deed-2800-410b-8950-7b402fd936b2" (UID: "60f8deed-2800-410b-8950-7b402fd936b2"). InnerVolumeSpecName "kube-api-access-rpbsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:44:58.609632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.609606    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca" pod "60f8deed-2800-410b-8950-7b402fd936b2" (UID: "60f8deed-2800-410b-8950-7b402fd936b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:44:58.709238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.709195    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbsw\" (UniqueName: \"kubernetes.io/projected/60f8deed-2800-410b-8950-7b402fd936b2-kube-api-access-rpbsw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.709238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.709235    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.709427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.709253    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8deed-2800-410b-8950-7b402fd936b2-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:44:58.807111 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.807069    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"]
Nov 01 08:44:58.811581 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.811552    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-685688c3-69c646f8fc-gbc5c"]
Nov 01 08:44:58.928877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:58.928794    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d17b28-1b1c-4117-b77c-a4f00b2c8786" path="/var/lib/kubelet/pods/05d17b28-1b1c-4117-b77c-a4f00b2c8786/volumes"
Nov 01 08:44:59.398481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.398453    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"
Nov 01 08:44:59.398481 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.398466    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx" event={"ID":"60f8deed-2800-410b-8950-7b402fd936b2","Type":"ContainerDied","Data":"57de3648097030954dd3eb061b91312a892c93f1d562a5ce94156fd80da36981"}
Nov 01 08:44:59.398977 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.398523    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57de3648097030954dd3eb061b91312a892c93f1d562a5ce94156fd80da36981"
Nov 01 08:44:59.400923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.400894    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"
Nov 01 08:44:59.401052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.400929    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8" event={"ID":"800340d0-8bfd-4159-9552-a63d9bbdf887","Type":"ContainerDied","Data":"dc762daa3be90a6256862c94f1d8d91945f2c772332fad8d57487652d62aba81"}
Nov 01 08:44:59.401052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.400993    4314 scope.go:122] "RemoveContainer" containerID="b4ac7e361fc4d94512b73a606fea8160d70600b5f5a974099cda7e903205ee7a"
Nov 01 08:44:59.554284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.554249    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"]
Nov 01 08:44:59.600424 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:44:59.600395    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-685688c3-849479c987-5r2c8"]
Nov 01 08:45:00.169713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.169676    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"]
Nov 01 08:45:00.170175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170156    4314 cpu_manager.go:414] "containerMap: removing container" podUID="05d17b28-1b1c-4117-b77c-a4f00b2c8786" containerName="payload-processing"
Nov 01 08:45:00.170278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170178    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d17b28-1b1c-4117-b77c-a4f00b2c8786" containerName="payload-processing"
Nov 01 08:45:00.170278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170196    4314 cpu_manager.go:414] "containerMap: removing container" podUID="60f8deed-2800-410b-8950-7b402fd936b2" containerName="revoke-keys"
Nov 01 08:45:00.170278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170203    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f8deed-2800-410b-8950-7b402fd936b2" containerName="revoke-keys"
Nov 01 08:45:00.170278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170256    4314 cpu_manager.go:414] "containerMap: removing container" podUID="800340d0-8bfd-4159-9552-a63d9bbdf887" containerName="payload-pre-processing"
Nov 01 08:45:00.170278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170264    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800340d0-8bfd-4159-9552-a63d9bbdf887" containerName="payload-pre-processing"
Nov 01 08:45:00.170504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170384    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="800340d0-8bfd-4159-9552-a63d9bbdf887" containerName="payload-pre-processing"
Nov 01 08:45:00.170504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170399    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d17b28-1b1c-4117-b77c-a4f00b2c8786" containerName="payload-processing"
Nov 01 08:45:00.170504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.170410    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f8deed-2800-410b-8950-7b402fd936b2" containerName="revoke-keys"
Nov 01 08:45:00.174868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.174847    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"]
Nov 01 08:45:00.174993 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.174965    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.178338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.178320    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"]
Nov 01 08:45:00.178450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.178418    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.181645 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.181626    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"]
Nov 01 08:45:00.181739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.181704    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.185759 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.185743    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.222671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.222642    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892045-hptmj\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.222791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.222685    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwz9\" (UniqueName: \"kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9\") pod \"maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.222861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.222813    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.222916 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.222866    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8ct\" (UniqueName: \"kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct\") pod \"maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.222916 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.222894    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlq8r\" (UniqueName: \"kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r\") pod \"maas-api-key-cleanup-29892045-hptmj\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.223015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.222950    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.223100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.223087    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.223163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.223120    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4ffn\" (UniqueName: \"kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn\") pod \"maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.223666 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.223647    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:45:00.223763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.223689    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:45:00.223817 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.223771    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:45:00.262441 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.262410    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"]
Nov 01 08:45:00.266324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.266294    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"]
Nov 01 08:45:00.309925 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.309900    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"]
Nov 01 08:45:00.328218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328188    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892045-hptmj\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.328389 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328300    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwz9\" (UniqueName: \"kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9\") pod \"maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.328464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328390    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.328464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328438    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8ct\" (UniqueName: \"kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct\") pod \"maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.328606 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328473    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlq8r\" (UniqueName: \"kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r\") pod \"maas-api-key-cleanup-29892045-hptmj\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.328606 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328550    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.328708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328646    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.328708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.328683    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4ffn\" (UniqueName: \"kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn\") pod \"maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.329073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.329049    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"]
Nov 01 08:45:00.330205 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.330178    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892045-hptmj\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.330310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.330201    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.330869 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.330846    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.330946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.330924    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.387001 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.386969    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8ct\" (UniqueName: \"kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct\") pod \"maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.387209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.387188    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4ffn\" (UniqueName: \"kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn\") pod \"maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.387316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.387292    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlq8r\" (UniqueName: \"kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r\") pod \"maas-api-key-cleanup-29892045-hptmj\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.387431 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.387413    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwz9\" (UniqueName: \"kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9\") pod \"maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.490128 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.490099    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:00.498253 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.498225    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:00.506305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.506273    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:00.513692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.513666    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:00.807147 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.807059    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:00.879155 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.879128    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"]
Nov 01 08:45:00.881202 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:00.881173    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb6581_0c45_43b3_817f_b11fe09329bb.slice/crio-7586e43a87627bdecea62b33a9fe7b20b06f8f94b36561948178307223d54868 WatchSource:0}: Error finding container 7586e43a87627bdecea62b33a9fe7b20b06f8f94b36561948178307223d54868: Status 404 returned error can't find the container with id 7586e43a87627bdecea62b33a9fe7b20b06f8f94b36561948178307223d54868
Nov 01 08:45:00.882657 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.882632    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"]
Nov 01 08:45:00.910953 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.910916    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"]
Nov 01 08:45:00.912730 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:00.912701    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60a8f54_1c2e_493b_8544_540e37f3a7cd.slice/crio-79c88933c294d6454635f5dca48bc4745413a8efdb1f3c753978bf5d2898a380 WatchSource:0}: Error finding container 79c88933c294d6454635f5dca48bc4745413a8efdb1f3c753978bf5d2898a380: Status 404 returned error can't find the container with id 79c88933c294d6454635f5dca48bc4745413a8efdb1f3c753978bf5d2898a380
Nov 01 08:45:00.937158 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:00.937115    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2907399_7d46_4fd9_868b_8fe0269c9e3e.slice/crio-f2dd98ed5a75333e2c58069473e19adf23fe7ac361472ff02d0e2db414dc96b2 WatchSource:0}: Error finding container f2dd98ed5a75333e2c58069473e19adf23fe7ac361472ff02d0e2db414dc96b2: Status 404 returned error can't find the container with id f2dd98ed5a75333e2c58069473e19adf23fe7ac361472ff02d0e2db414dc96b2
Nov 01 08:45:00.938337 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.938314    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800340d0-8bfd-4159-9552-a63d9bbdf887" path="/var/lib/kubelet/pods/800340d0-8bfd-4159-9552-a63d9bbdf887/volumes"
Nov 01 08:45:00.938681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:00.938667    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"]
Nov 01 08:45:01.034571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.034518    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"]
Nov 01 08:45:01.036115 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:01.036088    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f91ce34_a10e_4626_9cd6_20863e8cb1f3.slice/crio-d890f2f8bcac8bfcfbb268133e126f1feee0c4b5bc2937f1641abeaa4adc1667 WatchSource:0}: Error finding container d890f2f8bcac8bfcfbb268133e126f1feee0c4b5bc2937f1641abeaa4adc1667: Status 404 returned error can't find the container with id d890f2f8bcac8bfcfbb268133e126f1feee0c4b5bc2937f1641abeaa4adc1667
Nov 01 08:45:01.414372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.414337    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7" event={"ID":"1f91ce34-a10e-4626-9cd6-20863e8cb1f3","Type":"ContainerStarted","Data":"d890f2f8bcac8bfcfbb268133e126f1feee0c4b5bc2937f1641abeaa4adc1667"}
Nov 01 08:45:01.415999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.415973    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" event={"ID":"a2907399-7d46-4fd9-868b-8fe0269c9e3e","Type":"ContainerStarted","Data":"da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"}
Nov 01 08:45:01.416144 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.416016    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" event={"ID":"a2907399-7d46-4fd9-868b-8fe0269c9e3e","Type":"ContainerStarted","Data":"f2dd98ed5a75333e2c58069473e19adf23fe7ac361472ff02d0e2db414dc96b2"}
Nov 01 08:45:01.417563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.417543    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerStarted","Data":"f36a8a6b21def41d411678974420128893097967589f77366d2f8823b0fe5156"}
Nov 01 08:45:01.417666 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.417585    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerStarted","Data":"7586e43a87627bdecea62b33a9fe7b20b06f8f94b36561948178307223d54868"}
Nov 01 08:45:01.418718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.418699    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" event={"ID":"d60a8f54-1c2e-493b-8544-540e37f3a7cd","Type":"ContainerStarted","Data":"79c88933c294d6454635f5dca48bc4745413a8efdb1f3c753978bf5d2898a380"}
Nov 01 08:45:01.526634 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.526574    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" podStartSLOduration=1.526528232 podStartE2EDuration="1.526528232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:01.524968455 +0000 UTC m=+774.719363004" watchObservedRunningTime="2026-11-01 08:45:01.526528232 +0000 UTC m=+774.720922787"
Nov 01 08:45:01.632902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:01.632780    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" podStartSLOduration=1.632761804 podStartE2EDuration="1.632761804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:01.631578574 +0000 UTC m=+774.825973121" watchObservedRunningTime="2026-11-01 08:45:01.632761804 +0000 UTC m=+774.827156361"
Nov 01 08:45:02.064206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.064176    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:02.425588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.425551    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f91ce34-a10e-4626-9cd6-20863e8cb1f3" containerID="af87d751b24d8da9c48d3fb1d210ef399b94bce855c7b3e839167b43ab990375" exitCode=6
Nov 01 08:45:02.425786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.425625    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7" event={"ID":"1f91ce34-a10e-4626-9cd6-20863e8cb1f3","Type":"ContainerDied","Data":"af87d751b24d8da9c48d3fb1d210ef399b94bce855c7b3e839167b43ab990375"}
Nov 01 08:45:02.427459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.427351    4314 generic.go:356] "Generic (PLEG): container finished" podID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerID="da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94" exitCode=60
Nov 01 08:45:02.427459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.427405    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" event={"ID":"a2907399-7d46-4fd9-868b-8fe0269c9e3e","Type":"ContainerDied","Data":"da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"}
Nov 01 08:45:02.427777 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.427756    4314 scope.go:122] "RemoveContainer" containerID="da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"
Nov 01 08:45:02.429903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.429775    4314 generic.go:356] "Generic (PLEG): container finished" podID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerID="f36a8a6b21def41d411678974420128893097967589f77366d2f8823b0fe5156" exitCode=60
Nov 01 08:45:02.429903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.429856    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerDied","Data":"f36a8a6b21def41d411678974420128893097967589f77366d2f8823b0fe5156"}
Nov 01 08:45:02.430330 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.430190    4314 scope.go:122] "RemoveContainer" containerID="f36a8a6b21def41d411678974420128893097967589f77366d2f8823b0fe5156"
Nov 01 08:45:02.433350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.432629    4314 generic.go:356] "Generic (PLEG): container finished" podID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerID="4ccab54684bc6f96111965017937305b509b934c9530b9275966f4c824b69d8e" exitCode=60
Nov 01 08:45:02.433350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.432666    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" event={"ID":"d60a8f54-1c2e-493b-8544-540e37f3a7cd","Type":"ContainerDied","Data":"4ccab54684bc6f96111965017937305b509b934c9530b9275966f4c824b69d8e"}
Nov 01 08:45:02.433350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.432962    4314 scope.go:122] "RemoveContainer" containerID="4ccab54684bc6f96111965017937305b509b934c9530b9275966f4c824b69d8e"
Nov 01 08:45:02.437493 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.437457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:02.580476 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.580448    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:02.653650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.653616    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn\" (UniqueName: \"kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn\") pod \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") "
Nov 01 08:45:02.653845 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.653705    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca\") pod \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\" (UID: \"1f91ce34-a10e-4626-9cd6-20863e8cb1f3\") "
Nov 01 08:45:02.654070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.654042    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca" pod "1f91ce34-a10e-4626-9cd6-20863e8cb1f3" (UID: "1f91ce34-a10e-4626-9cd6-20863e8cb1f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:02.655764 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.655745    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn" pod "1f91ce34-a10e-4626-9cd6-20863e8cb1f3" (UID: "1f91ce34-a10e-4626-9cd6-20863e8cb1f3"). InnerVolumeSpecName "kube-api-access-j4ffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:02.726967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.726934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:02.754946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.754901    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4ffn\" (UniqueName: \"kubernetes.io/projected/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-kube-api-access-j4ffn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:02.754946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.754936    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f91ce34-a10e-4626-9cd6-20863e8cb1f3-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:02.789796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.789759    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx" podUID="659e39ad-f338-413b-be07-2e54eba7ee55" containerName="payload-pre-processing" containerID="cri-o://d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc" gracePeriod=25
Nov 01 08:45:02.959625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.959590    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"]
Nov 01 08:45:02.963116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:02.963092    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-685688c3-5s8zj"]
Nov 01 08:45:03.073578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.073542    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:03.105974 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.105944    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:45:03.161379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.161334    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume\") pod \"659e39ad-f338-413b-be07-2e54eba7ee55\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") "
Nov 01 08:45:03.161651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.161396    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm\" (UniqueName: \"kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm\") pod \"659e39ad-f338-413b-be07-2e54eba7ee55\" (UID: \"659e39ad-f338-413b-be07-2e54eba7ee55\") "
Nov 01 08:45:03.162329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.162296    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume" pod "659e39ad-f338-413b-be07-2e54eba7ee55" (UID: "659e39ad-f338-413b-be07-2e54eba7ee55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:03.164725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.164686    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm" pod "659e39ad-f338-413b-be07-2e54eba7ee55" (UID: "659e39ad-f338-413b-be07-2e54eba7ee55"). InnerVolumeSpecName "kube-api-access-952pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:03.236716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.236686    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:03.263446 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.263405    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659e39ad-f338-413b-be07-2e54eba7ee55-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:03.263446 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.263446    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952pm\" (UniqueName: \"kubernetes.io/projected/659e39ad-f338-413b-be07-2e54eba7ee55-kube-api-access-952pm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:03.313991 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.313946    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"]
Nov 01 08:45:03.440630 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.440596    4314 generic.go:356] "Generic (PLEG): container finished" podID="659e39ad-f338-413b-be07-2e54eba7ee55" containerID="d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc" exitCode=0
Nov 01 08:45:03.440803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.440644    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"
Nov 01 08:45:03.440803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.440681    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx" event={"ID":"659e39ad-f338-413b-be07-2e54eba7ee55","Type":"ContainerDied","Data":"d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc"}
Nov 01 08:45:03.440803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.440739    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx" event={"ID":"659e39ad-f338-413b-be07-2e54eba7ee55","Type":"ContainerDied","Data":"19c1a4b0f51446e35118a76517d6d0e24b3c450809d05b93a09f12e19917eabf"}
Nov 01 08:45:03.440803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.440761    4314 scope.go:122] "RemoveContainer" containerID="d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc"
Nov 01 08:45:03.442768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.442739    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerStarted","Data":"3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78"}
Nov 01 08:45:03.444785 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.444759    4314 generic.go:356] "Generic (PLEG): container finished" podID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerID="d687ff96aa881348e35f4f2f73fdf717ab80fc92131901c4d19836f8974196d8" exitCode=60
Nov 01 08:45:03.444891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.444792    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" event={"ID":"d60a8f54-1c2e-493b-8544-540e37f3a7cd","Type":"ContainerDied","Data":"d687ff96aa881348e35f4f2f73fdf717ab80fc92131901c4d19836f8974196d8"}
Nov 01 08:45:03.445091 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.445076    4314 scope.go:122] "RemoveContainer" containerID="d687ff96aa881348e35f4f2f73fdf717ab80fc92131901c4d19836f8974196d8"
Nov 01 08:45:03.445394 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:03.445361    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc_odh-ai-gateway-infra(d60a8f54-1c2e-493b-8544-540e37f3a7cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd"
Nov 01 08:45:03.446200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.446181    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"
Nov 01 08:45:03.446300 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.446185    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7" event={"ID":"1f91ce34-a10e-4626-9cd6-20863e8cb1f3","Type":"ContainerDied","Data":"d890f2f8bcac8bfcfbb268133e126f1feee0c4b5bc2937f1641abeaa4adc1667"}
Nov 01 08:45:03.447961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.447940    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" event={"ID":"a2907399-7d46-4fd9-868b-8fe0269c9e3e","Type":"ContainerStarted","Data":"ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4"}
Nov 01 08:45:03.465394 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.465368    4314 scope.go:122] "RemoveContainer" containerID="d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc"
Nov 01 08:45:03.465812 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:03.465781    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc\": container with ID starting with d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc not found: ID does not exist" containerID="d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc"
Nov 01 08:45:03.465898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.465823    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc"} err="failed to get container status \"d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc\": rpc error: code = NotFound desc = could not find container \"d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc\": container with ID starting with d7d33ffd4fbef73402720550d74c7e210dcd513d2e9c3fecad452cbdeeb5ffcc not found: ID does not exist"
Nov 01 08:45:03.465898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.465853    4314 scope.go:122] "RemoveContainer" containerID="4ccab54684bc6f96111965017937305b509b934c9530b9275966f4c824b69d8e"
Nov 01 08:45:03.537363 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.537341    4314 scope.go:122] "RemoveContainer" containerID="af87d751b24d8da9c48d3fb1d210ef399b94bce855c7b3e839167b43ab990375"
Nov 01 08:45:03.616088 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.616049    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:03.811244 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.811158    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"]
Nov 01 08:45:03.814926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.814891    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c25ae7a-5d6dbcc595-wzggx"]
Nov 01 08:45:03.852491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.852457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:03.916059 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.916022    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"]
Nov 01 08:45:03.918212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.918182    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-685688c3-29892045-wbqn7"]
Nov 01 08:45:03.998015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:03.997976    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:04.142578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.142462    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:04.178752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.178715    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"]
Nov 01 08:45:04.180796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.180778    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c25ae7a-zqrqn"]
Nov 01 08:45:04.373438 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.373404    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"]
Nov 01 08:45:04.373896 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.373881    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f91ce34-a10e-4626-9cd6-20863e8cb1f3" containerName="cleanup"
Nov 01 08:45:04.373940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.373898    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f91ce34-a10e-4626-9cd6-20863e8cb1f3" containerName="cleanup"
Nov 01 08:45:04.373940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.373925    4314 cpu_manager.go:414] "containerMap: removing container" podUID="659e39ad-f338-413b-be07-2e54eba7ee55" containerName="payload-pre-processing"
Nov 01 08:45:04.373940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.373931    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659e39ad-f338-413b-be07-2e54eba7ee55" containerName="payload-pre-processing"
Nov 01 08:45:04.374044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.374028    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f91ce34-a10e-4626-9cd6-20863e8cb1f3" containerName="cleanup"
Nov 01 08:45:04.374044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.374038    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="659e39ad-f338-413b-be07-2e54eba7ee55" containerName="payload-pre-processing"
Nov 01 08:45:04.378574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.378521    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.427267 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.427174    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-343207-openshift-default-dockercfg-j8ktj\""
Nov 01 08:45:04.434944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.434913    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"]
Nov 01 08:45:04.458644 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.458610    4314 generic.go:356] "Generic (PLEG): container finished" podID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerID="3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78" exitCode=60
Nov 01 08:45:04.458812 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.458698    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerDied","Data":"3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78"}
Nov 01 08:45:04.458812 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.458772    4314 scope.go:122] "RemoveContainer" containerID="f36a8a6b21def41d411678974420128893097967589f77366d2f8823b0fe5156"
Nov 01 08:45:04.459172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.459153    4314 scope.go:122] "RemoveContainer" containerID="3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78"
Nov 01 08:45:04.459462 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:04.459431    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892045-hptmj_odh-ai-gateway-infra(4ddb6581-0c45-43b3-817f-b11fe09329bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb"
Nov 01 08:45:04.460435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.460390    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup" containerID="cri-o://ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4" gracePeriod=30
Nov 01 08:45:04.460738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.460724    4314 scope.go:122] "RemoveContainer" containerID="d687ff96aa881348e35f4f2f73fdf717ab80fc92131901c4d19836f8974196d8"
Nov 01 08:45:04.460925 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:04.460907    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc_odh-ai-gateway-infra(d60a8f54-1c2e-493b-8544-540e37f3a7cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd"
Nov 01 08:45:04.463611 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.463578    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:04.476821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.476795    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbgl\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.476911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.476842    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.476911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.476873    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.477043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.477017    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.477090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.477076    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.477162 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.477148    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.477197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.477181    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.477231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.477208    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.477264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.477248    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578241    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578299    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578327    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578365    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578642    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbgl\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578709    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578741    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578748    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578829    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.578893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.578885    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.579064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.579042    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.579144 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.579087    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.579580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.579524    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.579756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.579737    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.581694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.581667    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.582053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.582030    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.636800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.636770    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.637176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.636871    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbgl\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.657689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.657649    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:04.692770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.692690    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:04.802725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.802687    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:04.928081 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.928047    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003182a4-0623-46c4-9f7c-774cfe50ab93" path="/var/lib/kubelet/pods/003182a4-0623-46c4-9f7c-774cfe50ab93/volumes"
Nov 01 08:45:04.928437 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.928424    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f91ce34-a10e-4626-9cd6-20863e8cb1f3" path="/var/lib/kubelet/pods/1f91ce34-a10e-4626-9cd6-20863e8cb1f3/volumes"
Nov 01 08:45:04.928770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.928758    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659e39ad-f338-413b-be07-2e54eba7ee55" path="/var/lib/kubelet/pods/659e39ad-f338-413b-be07-2e54eba7ee55/volumes"
Nov 01 08:45:04.929385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:04.929372    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b573fc43-28ce-4a17-901c-aa732f2a2d0c" path="/var/lib/kubelet/pods/b573fc43-28ce-4a17-901c-aa732f2a2d0c/volumes"
Nov 01 08:45:05.119047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.119021    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"]
Nov 01 08:45:05.121657 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:05.121627    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e08e935_62b5_4295_840b_6ba4f67c27b3.slice/crio-d6938f050bb445c4067aadda7c1b445a3257fb3db4997324b73cf608308df803 WatchSource:0}: Error finding container d6938f050bb445c4067aadda7c1b445a3257fb3db4997324b73cf608308df803: Status 404 returned error can't find the container with id d6938f050bb445c4067aadda7c1b445a3257fb3db4997324b73cf608308df803
Nov 01 08:45:05.123957 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.123923    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:05.124043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.123992    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:05.124043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.124023    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:05.331461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.331412    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:05.469733 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.469618    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" event={"ID":"0e08e935-62b5-4295-840b-6ba4f67c27b3","Type":"ContainerStarted","Data":"a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"}
Nov 01 08:45:05.470334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.469702    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" event={"ID":"0e08e935-62b5-4295-840b-6ba4f67c27b3","Type":"ContainerStarted","Data":"d6938f050bb445c4067aadda7c1b445a3257fb3db4997324b73cf608308df803"}
Nov 01 08:45:05.471727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.471710    4314 scope.go:122] "RemoveContainer" containerID="3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78"
Nov 01 08:45:05.471949 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:05.471929    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892045-hptmj_odh-ai-gateway-infra(4ddb6581-0c45-43b3-817f-b11fe09329bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb"
Nov 01 08:45:05.515161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.515125    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:05.584459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.584393    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podStartSLOduration=1.584363207 podStartE2EDuration="1.584363207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:05.58115981 +0000 UTC m=+778.775554364" watchObservedRunningTime="2026-11-01 08:45:05.584363207 +0000 UTC m=+778.778757765"
Nov 01 08:45:05.692959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:05.692919    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:06.017872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.017838    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"]
Nov 01 08:45:06.021743 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.021722    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.073008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.072976    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-751a73-openshift-default-dockercfg-qq2cf\""
Nov 01 08:45:06.084086 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.084042    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"]
Nov 01 08:45:06.096592 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096555    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.096768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096604    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.096768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096743    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.096839 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096792    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.096871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096863    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.096909 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096881    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.096944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.096928    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.097029 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.097002    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4b2x\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.097135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.097107    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.110748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.110706    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.197957    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198022    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198086    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198165    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198214    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198246    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198288    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198315    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4b2x\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.198443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.198385    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.202719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.201442    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.202719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.201459    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.202719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.201448    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.202719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.201839    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.202719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.201955    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.204648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.204617    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.204982 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.204954    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.256944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.256902    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.257080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.257019    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4b2x\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.335302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.335199    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:06.651156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.651067    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:06.699573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.699517    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:06.761512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.761484    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"]
Nov 01 08:45:06.763443 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:06.763416    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fac6601_db7c_4e32_84fc_5920fa3979b8.slice/crio-a12f77995e2973c0e1a0a6df3be4ac55e1b519f417b7525d1fb08d0a04a9ba13 WatchSource:0}: Error finding container a12f77995e2973c0e1a0a6df3be4ac55e1b519f417b7525d1fb08d0a04a9ba13: Status 404 returned error can't find the container with id a12f77995e2973c0e1a0a6df3be4ac55e1b519f417b7525d1fb08d0a04a9ba13
Nov 01 08:45:06.769959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.769920    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:06.770084 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.769995    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:06.770084 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:06.770031    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:07.485020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.484986    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" event={"ID":"8fac6601-db7c-4e32-84fc-5920fa3979b8","Type":"ContainerStarted","Data":"41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"}
Nov 01 08:45:07.485191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.485043    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" event={"ID":"8fac6601-db7c-4e32-84fc-5920fa3979b8","Type":"ContainerStarted","Data":"a12f77995e2973c0e1a0a6df3be4ac55e1b519f417b7525d1fb08d0a04a9ba13"}
Nov 01 08:45:07.485281 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.485248    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:07.486150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.486130    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:07.582340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.582306    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"]
Nov 01 08:45:07.627850 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.627794    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podStartSLOduration=2.627777622 podStartE2EDuration="2.627777622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:07.614004008 +0000 UTC m=+780.808398566" watchObservedRunningTime="2026-11-01 08:45:07.627777622 +0000 UTC m=+780.822172177"
Nov 01 08:45:07.829768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.829680    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:07.974015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:07.973979    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:08.270204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.270173    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:08.335886 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.335848    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:08.341208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.341182    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:08.479718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.479680    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"]
Nov 01 08:45:08.499552 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.482454    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-395735a2-lmhd5"]
Nov 01 08:45:08.499552 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.495793    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:08.499552 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.496878    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:45:08.856322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.855657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:08.928135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:08.928096    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5033b2-e7e4-4015-b797-649201e98482" path="/var/lib/kubelet/pods/ce5033b2-e7e4-4015-b797-649201e98482/volumes"
Nov 01 08:45:09.063246 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:09.063204    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:09.217347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:09.217265    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:09.935184 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:09.935154    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"]
Nov 01 08:45:09.935601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:09.935366    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy" containerID="cri-o://b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889" gracePeriod=30
Nov 01 08:45:09.945973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:09.945942    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:10.434288 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:10.434253    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:10.724753 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:10.724667    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:11.327063 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:11.327028    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:45:11.344752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:11.344721    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:11.643272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:11.643162    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:11.796564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:11.796488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:11.944870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:11.944776    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:12.312023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.311986    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:12.477235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.477206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:12.583261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.583154    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm" podUID="fda4d0b4-d80c-44b6-b078-f8360ace57a7" containerName="payload-pre-processing" containerID="cri-o://ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322" gracePeriod=25
Nov 01 08:45:12.862365 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.862330    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:45:12.975638 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.975601    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume\") pod \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") "
Nov 01 08:45:12.976740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.975670    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp\" (UniqueName: \"kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp\") pod \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\" (UID: \"fda4d0b4-d80c-44b6-b078-f8360ace57a7\") "
Nov 01 08:45:12.976740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.975922    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume" pod "fda4d0b4-d80c-44b6-b078-f8360ace57a7" (UID: "fda4d0b4-d80c-44b6-b078-f8360ace57a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:12.976740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.976141    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4d0b4-d80c-44b6-b078-f8360ace57a7-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:12.978421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:12.978394    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp" pod "fda4d0b4-d80c-44b6-b078-f8360ace57a7" (UID: "fda4d0b4-d80c-44b6-b078-f8360ace57a7"). InnerVolumeSpecName "kube-api-access-x6fsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:13.077177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.077141    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fsp\" (UniqueName: \"kubernetes.io/projected/fda4d0b4-d80c-44b6-b078-f8360ace57a7-kube-api-access-x6fsp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:13.521697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.521664    4314 generic.go:356] "Generic (PLEG): container finished" podID="fda4d0b4-d80c-44b6-b078-f8360ace57a7" containerID="ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322" exitCode=0
Nov 01 08:45:13.522159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.521720    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"
Nov 01 08:45:13.522159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.521744    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm" event={"ID":"fda4d0b4-d80c-44b6-b078-f8360ace57a7","Type":"ContainerDied","Data":"ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322"}
Nov 01 08:45:13.522159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.521797    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm" event={"ID":"fda4d0b4-d80c-44b6-b078-f8360ace57a7","Type":"ContainerDied","Data":"3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99"}
Nov 01 08:45:13.522159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.521821    4314 scope.go:122] "RemoveContainer" containerID="ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322"
Nov 01 08:45:13.530869 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.530847    4314 scope.go:122] "RemoveContainer" containerID="ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322"
Nov 01 08:45:13.531161 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:13.531141    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322\": container with ID starting with ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322 not found: ID does not exist" containerID="ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322"
Nov 01 08:45:13.531232 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.531169    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322"} err="failed to get container status \"ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322\": rpc error: code = NotFound desc = could not find container \"ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322\": container with ID starting with ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322 not found: ID does not exist"
Nov 01 08:45:13.676417 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.675212    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"]
Nov 01 08:45:13.720255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:13.720221    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f814ac10-678cd5d699-79xhm"]
Nov 01 08:45:14.928053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:14.928017    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda4d0b4-d80c-44b6-b078-f8360ace57a7" path="/var/lib/kubelet/pods/fda4d0b4-d80c-44b6-b078-f8360ace57a7/volumes"
Nov 01 08:45:15.186189 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.186121    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:45:15.299441 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299409    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299441 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299454    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299492    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299553    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299627    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299664    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299725    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299768    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.299865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299785    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:15.300020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299863    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token\") pod \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\" (UID: \"a6711ca6-c9f6-4178-b6f3-b206262f4b0c\") "
Nov 01 08:45:15.300020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299894    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:15.300020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299947    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:15.300020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.299997    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:15.300278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.300254    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.300278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.300281    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.300425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.300294    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.300425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.300295    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:15.300425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.300306    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.301984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.301956    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "kube-api-access-4nsvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:15.302076 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.302046    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:15.302357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.302332    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:15.302434 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.302335    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo" pod "a6711ca6-c9f6-4178-b6f3-b206262f4b0c" (UID: "a6711ca6-c9f6-4178-b6f3-b206262f4b0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 08:45:15.401596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.401512    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.401596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.401591    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nsvm\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-kube-api-access-4nsvm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.401868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.401616    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.401868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.401626    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.401868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.401634    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6711ca6-c9f6-4178-b6f3-b206262f4b0c-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:15.538079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.538046    4314 generic.go:356] "Generic (PLEG): container finished" podID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerID="b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889" exitCode=0
Nov 01 08:45:15.538257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.538099    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"
Nov 01 08:45:15.538257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.538136    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" event={"ID":"a6711ca6-c9f6-4178-b6f3-b206262f4b0c","Type":"ContainerDied","Data":"b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889"}
Nov 01 08:45:15.538257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.538190    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582" event={"ID":"a6711ca6-c9f6-4178-b6f3-b206262f4b0c","Type":"ContainerDied","Data":"26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0"}
Nov 01 08:45:15.538257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.538217    4314 scope.go:122] "RemoveContainer" containerID="b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889"
Nov 01 08:45:15.548013 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.547997    4314 scope.go:122] "RemoveContainer" containerID="3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"
Nov 01 08:45:15.556364 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.556344    4314 scope.go:122] "RemoveContainer" containerID="b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889"
Nov 01 08:45:15.556676 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:15.556658    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889\": container with ID starting with b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889 not found: ID does not exist" containerID="b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889"
Nov 01 08:45:15.556728 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.556686    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889"} err="failed to get container status \"b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889\": rpc error: code = NotFound desc = could not find container \"b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889\": container with ID starting with b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889 not found: ID does not exist"
Nov 01 08:45:15.556728 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.556705    4314 scope.go:122] "RemoveContainer" containerID="3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"
Nov 01 08:45:15.556983 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:15.556964    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210\": container with ID starting with 3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210 not found: ID does not exist" containerID="3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"
Nov 01 08:45:15.557023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.556989    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210"} err="failed to get container status \"3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210\": rpc error: code = NotFound desc = could not find container \"3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210\": container with ID starting with 3df99a8a07bae41eb705fdeff3eb5c419adb4c70386c2f00f07440a94fa49210 not found: ID does not exist"
Nov 01 08:45:15.717372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.717335    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"]
Nov 01 08:45:15.723658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.723629    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-395735a2-openshift-default-548dddf77f-zc582"]
Nov 01 08:45:15.806437 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.806353    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:15.921432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.921396    4314 scope.go:122] "RemoveContainer" containerID="d687ff96aa881348e35f4f2f73fdf717ab80fc92131901c4d19836f8974196d8"
Nov 01 08:45:15.921653 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:15.921499    4314 scope.go:122] "RemoveContainer" containerID="3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78"
Nov 01 08:45:16.546274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:16.546239    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerStarted","Data":"c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5"}
Nov 01 08:45:16.548030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:16.548007    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" event={"ID":"d60a8f54-1c2e-493b-8544-540e37f3a7cd","Type":"ContainerStarted","Data":"4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153"}
Nov 01 08:45:16.765090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:16.765030    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" podStartSLOduration=16.765007862 podStartE2EDuration="16.765007862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:16.761890778 +0000 UTC m=+789.956285346" watchObservedRunningTime="2026-11-01 08:45:16.765007862 +0000 UTC m=+789.959402417"
Nov 01 08:45:16.856782 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:16.856693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:16.930019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:16.929985    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" path="/var/lib/kubelet/pods/a6711ca6-c9f6-4178-b6f3-b206262f4b0c/volumes"
Nov 01 08:45:17.018242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.018208    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:17.385119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.385065    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:17.557236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.557202    4314 generic.go:356] "Generic (PLEG): container finished" podID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerID="c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5" exitCode=60
Nov 01 08:45:17.557771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.557281    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerDied","Data":"c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5"}
Nov 01 08:45:17.557771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.557349    4314 scope.go:122] "RemoveContainer" containerID="3bc5518fe62d02a7d2286b19d9acbbe6259b94ab02da866e3dd01b5119081c78"
Nov 01 08:45:17.557856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.557836    4314 scope.go:122] "RemoveContainer" containerID="c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5"
Nov 01 08:45:17.558176 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:17.558139    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892045-hptmj_odh-ai-gateway-infra(4ddb6581-0c45-43b3-817f-b11fe09329bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb"
Nov 01 08:45:17.559820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.559797    4314 generic.go:356] "Generic (PLEG): container finished" podID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerID="4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153" exitCode=60
Nov 01 08:45:17.559925 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.559873    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" event={"ID":"d60a8f54-1c2e-493b-8544-540e37f3a7cd","Type":"ContainerDied","Data":"4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153"}
Nov 01 08:45:17.560229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.560215    4314 scope.go:122] "RemoveContainer" containerID="4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153"
Nov 01 08:45:17.561041 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:17.560471    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc_odh-ai-gateway-infra(d60a8f54-1c2e-493b-8544-540e37f3a7cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd"
Nov 01 08:45:17.563719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.563699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:17.575239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.575217    4314 scope.go:122] "RemoveContainer" containerID="d687ff96aa881348e35f4f2f73fdf717ab80fc92131901c4d19836f8974196d8"
Nov 01 08:45:17.665711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.665624    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"]
Nov 01 08:45:17.710829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.710795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:17.775035 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:17.774999    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"]
Nov 01 08:45:18.710059 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.710031    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:18.754204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.754174    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:18.841898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.841813    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9\" (UniqueName: \"kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9\") pod \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") "
Nov 01 08:45:18.841898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.841852    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca\") pod \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\" (UID: \"d60a8f54-1c2e-493b-8544-540e37f3a7cd\") "
Nov 01 08:45:18.841898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.841891    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca\") pod \"4ddb6581-0c45-43b3-817f-b11fe09329bb\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") "
Nov 01 08:45:18.842186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.841991    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r\" (UniqueName: \"kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r\") pod \"4ddb6581-0c45-43b3-817f-b11fe09329bb\" (UID: \"4ddb6581-0c45-43b3-817f-b11fe09329bb\") "
Nov 01 08:45:18.842270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.842242    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca" pod "d60a8f54-1c2e-493b-8544-540e37f3a7cd" (UID: "d60a8f54-1c2e-493b-8544-540e37f3a7cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:18.842322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.842282    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca" pod "4ddb6581-0c45-43b3-817f-b11fe09329bb" (UID: "4ddb6581-0c45-43b3-817f-b11fe09329bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:18.842428 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.842413    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60a8f54-1c2e-493b-8544-540e37f3a7cd-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:18.842462 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.842431    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb6581-0c45-43b3-817f-b11fe09329bb-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:18.844009 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.843989    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r" pod "4ddb6581-0c45-43b3-817f-b11fe09329bb" (UID: "4ddb6581-0c45-43b3-817f-b11fe09329bb"). InnerVolumeSpecName "kube-api-access-nlq8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:18.844085 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.844055    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9" pod "d60a8f54-1c2e-493b-8544-540e37f3a7cd" (UID: "d60a8f54-1c2e-493b-8544-540e37f3a7cd"). InnerVolumeSpecName "kube-api-access-hjwz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:18.943949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.943883    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwz9\" (UniqueName: \"kubernetes.io/projected/d60a8f54-1c2e-493b-8544-540e37f3a7cd-kube-api-access-hjwz9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:18.943949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:18.943914    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlq8r\" (UniqueName: \"kubernetes.io/projected/4ddb6581-0c45-43b3-817f-b11fe09329bb-kube-api-access-nlq8r\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:19.575856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.575824    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"
Nov 01 08:45:19.576049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.575831    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj" event={"ID":"4ddb6581-0c45-43b3-817f-b11fe09329bb","Type":"ContainerDied","Data":"7586e43a87627bdecea62b33a9fe7b20b06f8f94b36561948178307223d54868"}
Nov 01 08:45:19.576049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.575980    4314 scope.go:122] "RemoveContainer" containerID="c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5"
Nov 01 08:45:19.577348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.577235    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"
Nov 01 08:45:19.577348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.577275    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc" event={"ID":"d60a8f54-1c2e-493b-8544-540e37f3a7cd","Type":"ContainerDied","Data":"79c88933c294d6454635f5dca48bc4745413a8efdb1f3c753978bf5d2898a380"}
Nov 01 08:45:19.585558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.585518    4314 scope.go:122] "RemoveContainer" containerID="4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153"
Nov 01 08:45:19.728130 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.728092    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"]
Nov 01 08:45:19.732201 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.732169    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892045-hptmj"]
Nov 01 08:45:19.833373 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.833289    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"]
Nov 01 08:45:19.837490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:19.837458    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f814ac10-29892045-rvzzc"]
Nov 01 08:45:20.029235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.029202    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"]
Nov 01 08:45:20.125587 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.125483    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"]
Nov 01 08:45:20.303625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.303544    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:20.457681 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.457592    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:20.600827 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.600795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:20.927623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.927590    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" path="/var/lib/kubelet/pods/4ddb6581-0c45-43b3-817f-b11fe09329bb/volumes"
Nov 01 08:45:20.928188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:20.928157    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" path="/var/lib/kubelet/pods/d60a8f54-1c2e-493b-8544-540e37f3a7cd/volumes"
Nov 01 08:45:21.911980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:21.911949    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:22.138281 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.138253    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:22.280653 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.280612    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:22.424187 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.424147    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:22.440054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.440017    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"]
Nov 01 08:45:22.487590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.487555    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f814ac10-fh9mx"]
Nov 01 08:45:22.491243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.491214    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"]
Nov 01 08:45:22.491488 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.491458    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" containerID="cri-o://9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837" gracePeriod=30
Nov 01 08:45:22.566784 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.566704    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:22.933893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:22.933800    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f8deed-2800-410b-8950-7b402fd936b2" path="/var/lib/kubelet/pods/60f8deed-2800-410b-8950-7b402fd936b2/volumes"
Nov 01 08:45:23.328460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.328406    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:23.815922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.815886    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"]
Nov 01 08:45:23.816430 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816410    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816431    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816443    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816451    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816484    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816492    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816509    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816516    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816528    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816549    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816563    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:23.816577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816570    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816590    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fda4d0b4-d80c-44b6-b078-f8360ace57a7" containerName="payload-pre-processing"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816597    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda4d0b4-d80c-44b6-b078-f8360ace57a7" containerName="payload-pre-processing"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816702    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda4d0b4-d80c-44b6-b078-f8360ace57a7" containerName="payload-pre-processing"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816717    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816729    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816737    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816749    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6711ca6-c9f6-4178-b6f3-b206262f4b0c" containerName="istio-proxy"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816759    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:23.817101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.816767    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:23.820275 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.820253    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.870574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.869160    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-54e01b-openshift-default-dockercfg-gln2n\""
Nov 01 08:45:23.896414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896375    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896574    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2q2n\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896597    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896755 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896619    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896794 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896768    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896810    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896833    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.896861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.896851    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.915776 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.915736    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:23.922167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.922137    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"]
Nov 01 08:45:23.998545 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998508    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.998767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998600    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.998767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998655    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2q2n\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.998767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998678    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.998767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998706    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.998767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998762    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.999025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998808    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.999025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998839    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:23.999025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.998866    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.001200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:23.999963    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.001200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.000519    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.001200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.000921    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.001200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.001157    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.002164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.002143    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.022038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.014346    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.022038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.014560    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.057640 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.057603    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.057802 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.057707    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2q2n\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.133927 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.133837    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:24.300170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.300096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:24.549601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.549572    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"]
Nov 01 08:45:24.554257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.554224    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:24.554346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.554291    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:24.554346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.554320    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:24.622739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:24.622710    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" event={"ID":"4fd04ace-2609-40d0-9a61-82b45be1f283","Type":"ContainerStarted","Data":"1f92b2df3a4fd821adac1ca1cf008940525f7d75541f91d8e1ea32f0f470ca80"}
Nov 01 08:45:25.029794 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.029745    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr" podUID="28ebf7f2-282f-41c5-bd4a-99b995980dfc" containerName="payload-processing" containerID="cri-o://9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d" gracePeriod=25
Nov 01 08:45:25.126236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.126170    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns" podUID="10c6301d-730f-4367-85d9-abd75b92bdea" containerName="payload-pre-processing" containerID="cri-o://4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e" gracePeriod=25
Nov 01 08:45:25.401555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.401499    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:45:25.410716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.410691    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:45:25.516650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.516610    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume\") pod \"10c6301d-730f-4367-85d9-abd75b92bdea\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") "
Nov 01 08:45:25.516836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.516671    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg\" (UniqueName: \"kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg\") pod \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") "
Nov 01 08:45:25.516836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.516790    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw\") pod \"10c6301d-730f-4367-85d9-abd75b92bdea\" (UID: \"10c6301d-730f-4367-85d9-abd75b92bdea\") "
Nov 01 08:45:25.516963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.516863    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume\") pod \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\" (UID: \"28ebf7f2-282f-41c5-bd4a-99b995980dfc\") "
Nov 01 08:45:25.517509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.517073    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume" pod "10c6301d-730f-4367-85d9-abd75b92bdea" (UID: "10c6301d-730f-4367-85d9-abd75b92bdea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:25.517509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.517425    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume" pod "28ebf7f2-282f-41c5-bd4a-99b995980dfc" (UID: "28ebf7f2-282f-41c5-bd4a-99b995980dfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:25.517509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.517469    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c6301d-730f-4367-85d9-abd75b92bdea-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:25.518996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.518965    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw" pod "10c6301d-730f-4367-85d9-abd75b92bdea" (UID: "10c6301d-730f-4367-85d9-abd75b92bdea"). InnerVolumeSpecName "kube-api-access-dncxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:25.519105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.519000    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg" pod "28ebf7f2-282f-41c5-bd4a-99b995980dfc" (UID: "28ebf7f2-282f-41c5-bd4a-99b995980dfc"). InnerVolumeSpecName "kube-api-access-k5nkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:25.618570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.618421    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5nkg\" (UniqueName: \"kubernetes.io/projected/28ebf7f2-282f-41c5-bd4a-99b995980dfc-kube-api-access-k5nkg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:25.618570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.618457    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/10c6301d-730f-4367-85d9-abd75b92bdea-kube-api-access-dncxw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:25.618570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.618472    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ebf7f2-282f-41c5-bd4a-99b995980dfc-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:25.629522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.629484    4314 generic.go:356] "Generic (PLEG): container finished" podID="28ebf7f2-282f-41c5-bd4a-99b995980dfc" containerID="9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d" exitCode=0
Nov 01 08:45:25.629722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.629583    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr" event={"ID":"28ebf7f2-282f-41c5-bd4a-99b995980dfc","Type":"ContainerDied","Data":"9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d"}
Nov 01 08:45:25.629722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.629696    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr" event={"ID":"28ebf7f2-282f-41c5-bd4a-99b995980dfc","Type":"ContainerDied","Data":"da0e69af24e83d159ee24d954693ce383e1a1b97969917ec831b1f07bc5ef248"}
Nov 01 08:45:25.629722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.629616    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"
Nov 01 08:45:25.629988 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.629716    4314 scope.go:122] "RemoveContainer" containerID="9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d"
Nov 01 08:45:25.631914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.631887    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" event={"ID":"4fd04ace-2609-40d0-9a61-82b45be1f283","Type":"ContainerStarted","Data":"75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"}
Nov 01 08:45:25.639061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.639028    4314 generic.go:356] "Generic (PLEG): container finished" podID="10c6301d-730f-4367-85d9-abd75b92bdea" containerID="4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e" exitCode=0
Nov 01 08:45:25.639195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.639127    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns" event={"ID":"10c6301d-730f-4367-85d9-abd75b92bdea","Type":"ContainerDied","Data":"4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e"}
Nov 01 08:45:25.639195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.639151    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"
Nov 01 08:45:25.639195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.639175    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns" event={"ID":"10c6301d-730f-4367-85d9-abd75b92bdea","Type":"ContainerDied","Data":"f89367adc57173da03a128dec3bbb59293fc1b5d489a24885524faf3f1289877"}
Nov 01 08:45:25.641685 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.641656    4314 scope.go:122] "RemoveContainer" containerID="9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d"
Nov 01 08:45:25.642025 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:25.641994    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d\": container with ID starting with 9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d not found: ID does not exist" containerID="9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d"
Nov 01 08:45:25.642108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.642038    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d"} err="failed to get container status \"9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d\": rpc error: code = NotFound desc = could not find container \"9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d\": container with ID starting with 9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d not found: ID does not exist"
Nov 01 08:45:25.642108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.642061    4314 scope.go:122] "RemoveContainer" containerID="4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e"
Nov 01 08:45:25.653036 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.653017    4314 scope.go:122] "RemoveContainer" containerID="4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e"
Nov 01 08:45:25.653323 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:25.653306    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e\": container with ID starting with 4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e not found: ID does not exist" containerID="4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e"
Nov 01 08:45:25.653393 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.653330    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e"} err="failed to get container status \"4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e\": rpc error: code = NotFound desc = could not find container \"4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e\": container with ID starting with 4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e not found: ID does not exist"
Nov 01 08:45:25.792391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.792354    4314 patch_prober.go:29] interesting pod/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 08:45:25.792567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.792414    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:45:25.793271 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.793229    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podStartSLOduration=2.793216263 podStartE2EDuration="2.793216263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:25.790933078 +0000 UTC m=+798.985327631" watchObservedRunningTime="2026-11-01 08:45:25.793216263 +0000 UTC m=+798.987610819"
Nov 01 08:45:25.814508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.814475    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:25.898395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.898313    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"]
Nov 01 08:45:25.904134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.904107    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0f765cd6-789b8fb5d5-b6mpr"]
Nov 01 08:45:25.979825 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:25.979788    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:26.005238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.005196    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"]
Nov 01 08:45:26.011563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.011506    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f765cd6-5654b697bc-5tjns"]
Nov 01 08:45:26.134972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.134931    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:26.140356 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.140326    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:26.192025 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:26.191951    4314 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/4b93e248901483941e2409179bad57e3a8611ddb9bed009c3b06f2f60bc3fd87/diff" to get inode usage: stat /var/lib/containers/storage/overlay/4b93e248901483941e2409179bad57e3a8611ddb9bed009c3b06f2f60bc3fd87/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_e2e-ait-395735a2-openshift-default-548dddf77f-zc582_a6711ca6-c9f6-4178-b6f3-b206262f4b0c/istio-proxy/1.log" to get inode usage: stat /var/log/pods/openshift-ingress_e2e-ait-395735a2-openshift-default-548dddf77f-zc582_a6711ca6-c9f6-4178-b6f3-b206262f4b0c/istio-proxy/1.log: no such file or directory
Nov 01 08:45:26.220139 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220103    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"]
Nov 01 08:45:26.220584 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220568    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220587    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220604    4314 cpu_manager.go:414] "containerMap: removing container" podUID="28ebf7f2-282f-41c5-bd4a-99b995980dfc" containerName="payload-processing"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220610    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebf7f2-282f-41c5-bd4a-99b995980dfc" containerName="payload-processing"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220623    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220628    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220638    4314 cpu_manager.go:414] "containerMap: removing container" podUID="10c6301d-730f-4367-85d9-abd75b92bdea" containerName="payload-pre-processing"
Nov 01 08:45:26.220642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220642    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c6301d-730f-4367-85d9-abd75b92bdea" containerName="payload-pre-processing"
Nov 01 08:45:26.220847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220712    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c6301d-730f-4367-85d9-abd75b92bdea" containerName="payload-pre-processing"
Nov 01 08:45:26.220847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220725    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddb6581-0c45-43b3-817f-b11fe09329bb" containerName="cleanup"
Nov 01 08:45:26.220847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220737    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebf7f2-282f-41c5-bd4a-99b995980dfc" containerName="payload-processing"
Nov 01 08:45:26.220847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.220745    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60a8f54-1c2e-493b-8544-540e37f3a7cd" containerName="cleanup"
Nov 01 08:45:26.225337 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.225318    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.275206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.274793    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b711ae-dockercfg-87dpc\""
Nov 01 08:45:26.275206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.274890    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b711ae\""
Nov 01 08:45:26.281338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.280155    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"]
Nov 01 08:45:26.327554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.327500    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.327766 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.327700    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqh2j\" (UniqueName: \"kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j\") pod \"payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.380493 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.380462    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"]
Nov 01 08:45:26.384316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.384292    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.428572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.428520    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqh2j\" (UniqueName: \"kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j\") pod \"payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.428749 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.428661    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.429320 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.429299    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.438833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.438803    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"]
Nov 01 08:45:26.485871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.485823    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqh2j\" (UniqueName: \"kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j\") pod \"payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.530170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.530131    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p6t\" (UniqueName: \"kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t\") pod \"payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.530377 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.530319    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.538012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.537981    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:26.631303 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.631271    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52p6t\" (UniqueName: \"kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t\") pod \"payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.631708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.631389    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.632074 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.632053    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.648557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.648496    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:26.649642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.649621    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:45:26.689350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.688668    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p6t\" (UniqueName: \"kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t\") pod \"payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.697092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.697062    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:26.907239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.907209    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"]
Nov 01 08:45:26.928998 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.928957    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c6301d-730f-4367-85d9-abd75b92bdea" path="/var/lib/kubelet/pods/10c6301d-730f-4367-85d9-abd75b92bdea/volumes"
Nov 01 08:45:26.929497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:26.929480    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ebf7f2-282f-41c5-bd4a-99b995980dfc" path="/var/lib/kubelet/pods/28ebf7f2-282f-41c5-bd4a-99b995980dfc/volumes"
Nov 01 08:45:27.061146 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.061117    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"]
Nov 01 08:45:27.530208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.530176    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:27.657070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.657041    4314 generic.go:356] "Generic (PLEG): container finished" podID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerID="9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837" exitCode=0
Nov 01 08:45:27.657595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.657113    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" event={"ID":"4eaabd03-3f6c-40ce-82f6-f73902fb6591","Type":"ContainerDied","Data":"9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837"}
Nov 01 08:45:27.657595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.657189    4314 scope.go:122] "RemoveContainer" containerID="36205918b4bbc0dce3f94e1acfb1e73b529a62398dd88eac498372e79cb03178"
Nov 01 08:45:27.658945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.658918    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg" event={"ID":"d834dc53-e017-409a-8d6a-64960a1bc6f2","Type":"ContainerStarted","Data":"074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e"}
Nov 01 08:45:27.659064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.658976    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg" event={"ID":"d834dc53-e017-409a-8d6a-64960a1bc6f2","Type":"ContainerStarted","Data":"8131fd8f0c9f49a692da9e32ac2366fd8cc6964ec6d7cb1a060d6b4185b17731"}
Nov 01 08:45:27.659125 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.659095    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:27.660560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.660524    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr" event={"ID":"dfb92746-a22a-4898-a45d-de0a9935931c","Type":"ContainerStarted","Data":"8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002"}
Nov 01 08:45:27.660671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.660581    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr" event={"ID":"dfb92746-a22a-4898-a45d-de0a9935931c","Type":"ContainerStarted","Data":"c8f8b3e1d4f538c1c210f69fce38a81f78b73f599098b3248eddeb1a3a08eacb"}
Nov 01 08:45:27.685739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.685703    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:27.758148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.758082    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:45:27.769005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.768955    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg" podStartSLOduration=1.768941575 podStartE2EDuration="1.768941575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:27.766614025 +0000 UTC m=+800.961008578" watchObservedRunningTime="2026-11-01 08:45:27.768941575 +0000 UTC m=+800.963336129"
Nov 01 08:45:27.831135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.831095    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:27.875495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.875440    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr" podStartSLOduration=1.875425312 podStartE2EDuration="1.875425312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:27.873801682 +0000 UTC m=+801.068196237" watchObservedRunningTime="2026-11-01 08:45:27.875425312 +0000 UTC m=+801.069819928"
Nov 01 08:45:27.944643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944602    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944653    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944743    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944764    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944781    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944810    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944847    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.944902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944889    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.945214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944900    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:27.945214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.944973    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy\") pod \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\" (UID: \"4eaabd03-3f6c-40ce-82f6-f73902fb6591\") "
Nov 01 08:45:27.945319 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945211    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:27.945319 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945224    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:27.945319 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945243    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:27.945463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945400    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:27.945463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945419    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:27.945463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945434    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:27.945463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945449    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:27.945696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.945503    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:27.947045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.947015    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "kube-api-access-blqjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:27.947159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.947123    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 08:45:27.947375 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.947359    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:27.947419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:27.947371    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token" pod "4eaabd03-3f6c-40ce-82f6-f73902fb6591" (UID: "4eaabd03-3f6c-40ce-82f6-f73902fb6591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:28.046523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.046483    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:28.046523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.046514    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqjf\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-kube-api-access-blqjf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:28.046745 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.046556    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:28.046745 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.046569    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:28.046745 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.046581    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eaabd03-3f6c-40ce-82f6-f73902fb6591-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:28.667478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.667431    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn" event={"ID":"4eaabd03-3f6c-40ce-82f6-f73902fb6591","Type":"ContainerDied","Data":"c6776587a9c961bb377f1850108c36995a378e0fe49763f1e2f9c73020bbda9c"}
Nov 01 08:45:28.667945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.667498    4314 scope.go:122] "RemoveContainer" containerID="9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837"
Nov 01 08:45:28.667945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.667457    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"
Nov 01 08:45:28.667945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.667931    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:28.821031 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.820999    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"]
Nov 01 08:45:28.867191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.867161    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn"]
Nov 01 08:45:28.927149 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:28.927070    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" path="/var/lib/kubelet/pods/4eaabd03-3f6c-40ce-82f6-f73902fb6591/volumes"
Nov 01 08:45:29.822432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:29.822399    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:30.004370 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:30.004339    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:30.146587 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:30.146481    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:32.803274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:32.803208    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:34.241928 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:34.241895    4314 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/b9ed036f76ed82d3b462ac1cc9cb321a14132fb19998380d10a3d83cfd662e72/diff" to get inode usage: stat /var/lib/containers/storage/overlay/b9ed036f76ed82d3b462ac1cc9cb321a14132fb19998380d10a3d83cfd662e72/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn_4eaabd03-3f6c-40ce-82f6-f73902fb6591/istio-proxy/1.log" to get inode usage: stat /var/log/pods/openshift-ingress_e2e-migrate-f814ac10-gw-openshift-default-64d99497c-n4hnn_4eaabd03-3f6c-40ce-82f6-f73902fb6591/istio-proxy/1.log: no such file or directory
Nov 01 08:45:34.513684 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:34.513647    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60a8f54_1c2e_493b_8544_540e37f3a7cd.slice/crio-4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153.scope WatchSource:0}: Error finding container 4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153: Status 404 returned error can't find the container with id 4c846d387084c249a9348023473c2948a695ff0a5081e54001cea686ca4e9153
Nov 01 08:45:34.514047 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:34.514015    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb6581_0c45_43b3_817f_b11fe09329bb.slice/crio-c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5.scope WatchSource:0}: Error finding container c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5: Status 404 returned error can't find the container with id c1859f51c1c79eaa0b3681bb2d3ac02fb2f09ca4cec6d293f024fa329eb39fe5
Nov 01 08:45:34.531355 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:34.531270    4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-conmon-4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebf7f2_282f_41c5_bd4a_99b995980dfc.slice/crio-conmon-9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60a8f54_1c2e_493b_8544_540e37f3a7cd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-conmon-ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb6581_0c45_43b3_817f_b11fe09329bb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebf7f2_282f_41c5_bd4a_99b995980dfc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-conmon-b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2907399_7d46_4fd9_868b_8fe0269c9e3e.slice/crio-conmon-ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 08:45:34.531560 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:34.531400    4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-conmon-ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2907399_7d46_4fd9_868b_8fe0269c9e3e.slice/crio-f2dd98ed5a75333e2c58069473e19adf23fe7ac361472ff02d0e2db414dc96b2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-26ef97317fc6c73dcccf2410b58106d2fe2c44e2813eb62da6c47b25824218e0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebf7f2_282f_41c5_bd4a_99b995980dfc.slice/crio-conmon-9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-3cd8df052693e98ab62004d2b941cc86dcb73a9682e35f9ec9e2a4fff236aa99\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb6581_0c45_43b3_817f_b11fe09329bb.slice/crio-7586e43a87627bdecea62b33a9fe7b20b06f8f94b36561948178307223d54868\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-conmon-b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-conmon-4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6711ca6_c9f6_4178_b6f3_b206262f4b0c.slice/crio-b3916269a0072c637650e9dd80dcb2b03359c081d429339e7029f910c6225889.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice/crio-c6776587a9c961bb377f1850108c36995a378e0fe49763f1e2f9c73020bbda9c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60a8f54_1c2e_493b_8544_540e37f3a7cd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2907399_7d46_4fd9_868b_8fe0269c9e3e.slice/crio-ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4d0b4_d80c_44b6_b078_f8360ace57a7.slice/crio-ddca9f962dcf60eedb5654d6d6d2b3afbb677ec832e6635ec9a048dad17e0322.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebf7f2_282f_41c5_bd4a_99b995980dfc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb6581_0c45_43b3_817f_b11fe09329bb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice/crio-conmon-9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2907399_7d46_4fd9_868b_8fe0269c9e3e.slice/crio-conmon-ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 08:45:34.531709 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:34.531660    4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaabd03_3f6c_40ce_82f6_f73902fb6591.slice/crio-9c66aab694af2afa4837da2b1da699a6eae560c0b32529dbd067133e9c130837.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-f89367adc57173da03a128dec3bbb59293fc1b5d489a24885524faf3f1289877\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebf7f2_282f_41c5_bd4a_99b995980dfc.slice/crio-conmon-9673219f9341b5e0c37df7aa340428bf2a85b975ad15a395499643e6b7ae865d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6301d_730f_4367_85d9_abd75b92bdea.slice/crio-4145a2bdcbf72d3538a471ea1e58095e64be48e2870c85b6c6fdea9b7636536e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2907399_7d46_4fd9_868b_8fe0269c9e3e.slice/crio-ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 08:45:34.669137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.669109    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:34.671502 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.671474    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:34.702315 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.702282    4314 generic.go:356] "Generic (PLEG): container finished" podID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerID="ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4" exitCode=137
Nov 01 08:45:34.702490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.702336    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"
Nov 01 08:45:34.702490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.702342    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" event={"ID":"a2907399-7d46-4fd9-868b-8fe0269c9e3e","Type":"ContainerDied","Data":"ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4"}
Nov 01 08:45:34.702490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.702384    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp" event={"ID":"a2907399-7d46-4fd9-868b-8fe0269c9e3e","Type":"ContainerDied","Data":"f2dd98ed5a75333e2c58069473e19adf23fe7ac361472ff02d0e2db414dc96b2"}
Nov 01 08:45:34.702490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.702406    4314 scope.go:122] "RemoveContainer" containerID="ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4"
Nov 01 08:45:34.711361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.711336    4314 scope.go:122] "RemoveContainer" containerID="da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"
Nov 01 08:45:34.713723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.713695    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct\" (UniqueName: \"kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct\") pod \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") "
Nov 01 08:45:34.715798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.715776    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct" pod "a2907399-7d46-4fd9-868b-8fe0269c9e3e" (UID: "a2907399-7d46-4fd9-868b-8fe0269c9e3e"). InnerVolumeSpecName "kube-api-access-6v8ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:34.719973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.719953    4314 scope.go:122] "RemoveContainer" containerID="ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4"
Nov 01 08:45:34.720328 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:34.720302    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4\": container with ID starting with ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4 not found: ID does not exist" containerID="ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4"
Nov 01 08:45:34.720395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.720343    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4"} err="failed to get container status \"ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4\": rpc error: code = NotFound desc = could not find container \"ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4\": container with ID starting with ade16cc616f76dcd14d12dc6f53800f94824eb1c0ba5e755db303d7ff561acb4 not found: ID does not exist"
Nov 01 08:45:34.720395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.720366    4314 scope.go:122] "RemoveContainer" containerID="da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"
Nov 01 08:45:34.720700 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:34.720681    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94\": container with ID starting with da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94 not found: ID does not exist" containerID="da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"
Nov 01 08:45:34.720761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.720707    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94"} err="failed to get container status \"da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94\": rpc error: code = NotFound desc = could not find container \"da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94\": container with ID starting with da4f18fad59e35fce86c22fb6af1cb66f9e7b82f78e4705f14bfb878d2868f94 not found: ID does not exist"
Nov 01 08:45:34.814552 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.814450    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca\") pod \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\" (UID: \"a2907399-7d46-4fd9-868b-8fe0269c9e3e\") "
Nov 01 08:45:34.814880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.814853    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca" pod "a2907399-7d46-4fd9-868b-8fe0269c9e3e" (UID: "a2907399-7d46-4fd9-868b-8fe0269c9e3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:34.814921 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.814870    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8ct\" (UniqueName: \"kubernetes.io/projected/a2907399-7d46-4fd9-868b-8fe0269c9e3e-kube-api-access-6v8ct\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:34.915855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.915822    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2907399-7d46-4fd9-868b-8fe0269c9e3e-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:34.938509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:34.938473    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:35.113349 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.113250    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"]
Nov 01 08:45:35.117083 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.117050    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-395735a2-29892045-phsmp"]
Nov 01 08:45:35.136892 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.136846    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"]
Nov 01 08:45:35.137645 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137615    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup"
Nov 01 08:45:35.137645 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137644    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup"
Nov 01 08:45:35.137831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137663    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy"
Nov 01 08:45:35.137831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137672    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy"
Nov 01 08:45:35.137831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137699    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup"
Nov 01 08:45:35.137831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137707    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup"
Nov 01 08:45:35.137831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137822    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup"
Nov 01 08:45:35.138065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137834    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy"
Nov 01 08:45:35.138065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.137848    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" containerName="cleanup"
Nov 01 08:45:35.145555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.144556    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.196189 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.196150    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-54e01b\""
Nov 01 08:45:35.196461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.196440    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-54e01b-dockercfg-8c7pv\""
Nov 01 08:45:35.219610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.218302    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.219610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.218357    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzmb\" (UniqueName: \"kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb\") pod \"payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.225920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.224772    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"]
Nov 01 08:45:35.284032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.283995    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"]
Nov 01 08:45:35.284620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.284473    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy"
Nov 01 08:45:35.284620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.284486    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy"
Nov 01 08:45:35.284620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.284579    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eaabd03-3f6c-40ce-82f6-f73902fb6591" containerName="istio-proxy"
Nov 01 08:45:35.288153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.288133    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.318268 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.318231    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"]
Nov 01 08:45:35.318933 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.318909    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.319001 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.318983    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.319054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.319019    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzmb\" (UniqueName: \"kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb\") pod \"payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.319099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.319075    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kncbn\" (UniqueName: \"kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn\") pod \"payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.319761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.319736    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.385414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.385309    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzmb\" (UniqueName: \"kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb\") pod \"payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.420021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.419977    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncbn\" (UniqueName: \"kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn\") pod \"payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.420211 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.420073    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.420824 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.420802    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.479780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.478779    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:35.479780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.479389    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncbn\" (UniqueName: \"kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn\") pod \"payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.602487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.602450    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:35.848611 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.848584    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"]
Nov 01 08:45:35.850364 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:35.850338    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c5850c_149f_4021_aa38_43468f0e0dfe.slice/crio-a5a0ae9b74a5cdc5e6e19b99b6e47c9b0b5752067f47ef9496640576008b6d39 WatchSource:0}: Error finding container a5a0ae9b74a5cdc5e6e19b99b6e47c9b0b5752067f47ef9496640576008b6d39: Status 404 returned error can't find the container with id a5a0ae9b74a5cdc5e6e19b99b6e47c9b0b5752067f47ef9496640576008b6d39
Nov 01 08:45:35.982748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:35.982696    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"]
Nov 01 08:45:35.988744 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:35.988711    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29daa32_340d_4b9f_a146_c32c26c3ec03.slice/crio-1c8aa30613b3bae60a214959dff22a2c6bf1e147031baf7b1aae59aa495f2e76 WatchSource:0}: Error finding container 1c8aa30613b3bae60a214959dff22a2c6bf1e147031baf7b1aae59aa495f2e76: Status 404 returned error can't find the container with id 1c8aa30613b3bae60a214959dff22a2c6bf1e147031baf7b1aae59aa495f2e76
Nov 01 08:45:36.719414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.719376    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" event={"ID":"53c5850c-149f-4021-aa38-43468f0e0dfe","Type":"ContainerStarted","Data":"d1190265b9e3ddcc1ac199e9a8fc19abcb13794c15ef751bbaf70e9dc54002f7"}
Nov 01 08:45:36.719875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.719439    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" event={"ID":"53c5850c-149f-4021-aa38-43468f0e0dfe","Type":"ContainerStarted","Data":"a5a0ae9b74a5cdc5e6e19b99b6e47c9b0b5752067f47ef9496640576008b6d39"}
Nov 01 08:45:36.719875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.719467    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:36.720898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.720875    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" event={"ID":"e29daa32-340d-4b9f-a146-c32c26c3ec03","Type":"ContainerStarted","Data":"b3ec34d0a0b4e9e58e53519249c69081318931e0047ec8aca290ee3827989dc4"}
Nov 01 08:45:36.720979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.720916    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" event={"ID":"e29daa32-340d-4b9f-a146-c32c26c3ec03","Type":"ContainerStarted","Data":"1c8aa30613b3bae60a214959dff22a2c6bf1e147031baf7b1aae59aa495f2e76"}
Nov 01 08:45:36.720979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.720948    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:36.831120 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.831052    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" podStartSLOduration=1.831038314 podStartE2EDuration="1.831038314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:36.829717784 +0000 UTC m=+810.024112340" watchObservedRunningTime="2026-11-01 08:45:36.831038314 +0000 UTC m=+810.025432868"
Nov 01 08:45:36.927922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.927891    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2907399-7d46-4fd9-868b-8fe0269c9e3e" path="/var/lib/kubelet/pods/a2907399-7d46-4fd9-868b-8fe0269c9e3e/volumes"
Nov 01 08:45:36.939205 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:36.939156    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" podStartSLOduration=1.939140214 podStartE2EDuration="1.939140214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:36.938264858 +0000 UTC m=+810.132659414" watchObservedRunningTime="2026-11-01 08:45:36.939140214 +0000 UTC m=+810.133534769"
Nov 01 08:45:37.552554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:37.552471    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:37.717725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:37.717686    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:37.878289 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:37.878084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:38.683485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:38.683451    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 08:45:39.683549 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:39.683507    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 08:45:41.323472 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:41.323437    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:45:41.896909 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:41.896878    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:43.452470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:43.452434    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:43.671217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:43.671182    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:43.817357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:43.817322    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:44.046346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.046308    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"]
Nov 01 08:45:44.046629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.046601    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" containerID="cri-o://fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593" gracePeriod=30
Nov 01 08:45:44.060934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.060905    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:44.765029 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.764998    4314 generic.go:356] "Generic (PLEG): container finished" podID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerID="a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae" exitCode=0
Nov 01 08:45:44.765490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.765067    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" event={"ID":"0e08e935-62b5-4295-840b-6ba4f67c27b3","Type":"ContainerDied","Data":"a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"}
Nov 01 08:45:44.765490 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.765459    4314 scope.go:122] "RemoveContainer" containerID="a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"
Nov 01 08:45:44.767160 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.767126    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:44.767256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.767200    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:44.767256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.767233    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:45:44.853944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:44.853906    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:45.494513 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:45.494464    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:45.670677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:45.670284    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:45.789087 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:45.789000    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" event={"ID":"0e08e935-62b5-4295-840b-6ba4f67c27b3","Type":"ContainerStarted","Data":"a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627"}
Nov 01 08:45:46.693321 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:46.693285    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:46.693651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:46.693622    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 08:45:46.693807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:46.693669    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 08:45:47.694300 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:47.694268    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 08:45:47.694741 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:47.694319    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 08:45:47.729614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:47.729586    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 08:45:47.729791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:47.729681    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 08:45:48.684233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:48.684188    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:48.693580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:48.693508    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 08:45:48.693781 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:48.693603    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 08:45:48.834246 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:48.834205    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:48.983056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:48.983020    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:49.344522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.344495    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:45:49.526129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526043    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526094    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526121    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526150    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526188    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526267    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526312    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526347    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526363    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526396    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.526415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526405    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo\") pod \"b3f7e959-1747-46a6-8b9e-66191451ee20\" (UID: \"b3f7e959-1747-46a6-8b9e-66191451ee20\") "
Nov 01 08:45:49.526826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526696    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.526826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526796    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:49.526941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526839    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.526941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526858    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.526941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.526885    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.527135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.527116    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.529030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.528985    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 08:45:49.529030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.529009    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:49.529224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.529091    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "kube-api-access-prscb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:49.529224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.529086    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy" pod "b3f7e959-1747-46a6-8b9e-66191451ee20" (UID: "b3f7e959-1747-46a6-8b9e-66191451ee20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.628113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.628078    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.628113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.628115    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prscb\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-kube-api-access-prscb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.628339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.628131    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.628339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.628143    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.628339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.628155    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f7e959-1747-46a6-8b9e-66191451ee20-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.628339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.628167    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f7e959-1747-46a6-8b9e-66191451ee20-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.693987 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.693955    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 08:45:49.694176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.694006    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 08:45:49.814651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.814570    4314 generic.go:356] "Generic (PLEG): container finished" podID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerID="fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593" exitCode=0
Nov 01 08:45:49.814651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.814624    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"
Nov 01 08:45:49.814831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.814627    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" event={"ID":"b3f7e959-1747-46a6-8b9e-66191451ee20","Type":"ContainerDied","Data":"fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593"}
Nov 01 08:45:49.814831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.814762    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" event={"ID":"b3f7e959-1747-46a6-8b9e-66191451ee20","Type":"ContainerDied","Data":"a79bce1865b221e1d85e19e24466b3ffaa0fdfb1b3039898093a4c814647a126"}
Nov 01 08:45:49.814831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.814791    4314 scope.go:122] "RemoveContainer" containerID="fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593"
Nov 01 08:45:49.831118 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.831097    4314 scope.go:122] "RemoveContainer" containerID="6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"
Nov 01 08:45:49.839623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.839597    4314 scope.go:122] "RemoveContainer" containerID="fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593"
Nov 01 08:45:49.839977 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:49.839881    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593\": container with ID starting with fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593 not found: ID does not exist" containerID="fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593"
Nov 01 08:45:49.839977 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.839907    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593"} err="failed to get container status \"fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593\": rpc error: code = NotFound desc = could not find container \"fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593\": container with ID starting with fe49f88757c5870cec4d845d87070b9acb08f80af7e6c5d1120aa94f701a8593 not found: ID does not exist"
Nov 01 08:45:49.839977 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.839923    4314 scope.go:122] "RemoveContainer" containerID="6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"
Nov 01 08:45:49.840196 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:45:49.840169    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9\": container with ID starting with 6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9 not found: ID does not exist" containerID="6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"
Nov 01 08:45:49.840240 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.840202    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9"} err="failed to get container status \"6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9\": rpc error: code = NotFound desc = could not find container \"6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9\": container with ID starting with 6bbbf18715505f4881e4124afc9acac641e02a0b002220890308df3421ecd6d9 not found: ID does not exist"
Nov 01 08:45:49.985928 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.985879    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"]
Nov 01 08:45:49.987810 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:49.987783    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628"]
Nov 01 08:45:50.289792 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:50.289755    4314 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:45:50.289969 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:50.289819    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-rx628" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:45:50.693938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:50.693842    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 08:45:50.693938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:50.693909    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 08:45:50.927410 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:50.927368    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" path="/var/lib/kubelet/pods/b3f7e959-1747-46a6-8b9e-66191451ee20/volumes"
Nov 01 08:45:51.088247 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:51.088147    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:51.305858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:51.305828    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:51.455376 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:51.455299    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:51.697265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:51.697236    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:51.697621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:51.697603    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:51.698236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:51.698216    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 08:45:53.738275 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:53.738207    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:53.980724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:53.980691    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:54.342871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:54.342836    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:54.493242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:54.493156    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:54.642107 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:54.641979    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:55.939066 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:55.939019    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:56.100850 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.100795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:56.502177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502145    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"]
Nov 01 08:45:56.502682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502663    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy"
Nov 01 08:45:56.502785 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502681    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy"
Nov 01 08:45:56.502785 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502724    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy"
Nov 01 08:45:56.502785 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502732    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy"
Nov 01 08:45:56.502940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502848    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy"
Nov 01 08:45:56.502940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.502865    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f7e959-1747-46a6-8b9e-66191451ee20" containerName="istio-proxy"
Nov 01 08:45:56.506080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.506060    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.560859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.560826    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"]
Nov 01 08:45:56.595117 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.595078    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gth5\" (UniqueName: \"kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5\") pod \"payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.595298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.595128    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.603566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.603518    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dfdd13\""
Nov 01 08:45:56.603726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.603612    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dfdd13-dockercfg-6zpl8\""
Nov 01 08:45:56.696359 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.696325    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gth5\" (UniqueName: \"kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5\") pod \"payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.696516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.696366    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.697081 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.697061    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.753602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.753499    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gth5\" (UniqueName: \"kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5\") pod \"payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:56.818671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:56.818634    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:57.181050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.181025    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"]
Nov 01 08:45:57.183148 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:57.183121    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddb67ed_071b_4181_890f_56355834104f.slice/crio-88c72a992c13e9d7d8eaa495bcb4a9f93cdf87cb5378a4a0479fde35bc7a6589 WatchSource:0}: Error finding container 88c72a992c13e9d7d8eaa495bcb4a9f93cdf87cb5378a4a0479fde35bc7a6589: Status 404 returned error can't find the container with id 88c72a992c13e9d7d8eaa495bcb4a9f93cdf87cb5378a4a0479fde35bc7a6589
Nov 01 08:45:57.478225 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.478146    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"]
Nov 01 08:45:57.481747 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.481731    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.514730 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.514700    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.514895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.514739    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.514895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.514763    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.514987 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.514925    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.515043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.514982    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqhl\" (UniqueName: \"kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.515149 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.515127    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.533468 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.533440    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"openshift-service-ca.crt\""
Nov 01 08:45:57.568513 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.568479    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:57.579485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.579453    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"kube-root-ca.crt\""
Nov 01 08:45:57.579485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.579461    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"default-dockercfg-f8cwt\""
Nov 01 08:45:57.579485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.579461    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"simulator-w2-343207-kserve-self-signed-certs\""
Nov 01 08:45:57.581806 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.581788    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"]
Nov 01 08:45:57.617680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.616643    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.617680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.616701    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.617680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.616732    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.617680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.616758    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.617680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.616864    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.617680 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.616911    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqhl\" (UniqueName: \"kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.619604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.618960    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.619604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.619231    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.619604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.619551    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.623057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.623029    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.624235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.624209    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.676588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.676550    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqhl\" (UniqueName: \"kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl\") pod \"simulator-w2-343207-kserve-985dd9f64-d6t2n\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") " pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.793198 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.793161    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:45:57.863135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.863095    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" event={"ID":"dddb67ed-071b-4181-890f-56355834104f","Type":"ContainerStarted","Data":"467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8"}
Nov 01 08:45:57.863292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.863168    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" event={"ID":"dddb67ed-071b-4181-890f-56355834104f","Type":"ContainerStarted","Data":"88c72a992c13e9d7d8eaa495bcb4a9f93cdf87cb5378a4a0479fde35bc7a6589"}
Nov 01 08:45:57.863292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.863201    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:45:57.973907 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:57.973853    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" podStartSLOduration=1.973837692 podStartE2EDuration="1.973837692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:57.971598071 +0000 UTC m=+831.165992619" watchObservedRunningTime="2026-11-01 08:45:57.973837692 +0000 UTC m=+831.168232246"
Nov 01 08:45:58.159050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:58.159023    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"]
Nov 01 08:45:58.161640 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:45:58.161611    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bd618d9_cd13_4ab2_998b_a23c6ac10545.slice/crio-ef491f549ae3888b4db699eb03bae9a254ac37f9aa3e7cae30abb1691c3c1682 WatchSource:0}: Error finding container ef491f549ae3888b4db699eb03bae9a254ac37f9aa3e7cae30abb1691c3c1682: Status 404 returned error can't find the container with id ef491f549ae3888b4db699eb03bae9a254ac37f9aa3e7cae30abb1691c3c1682
Nov 01 08:45:58.902078 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:58.901177    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" event={"ID":"2bd618d9-cd13-4ab2-998b-a23c6ac10545","Type":"ContainerStarted","Data":"da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696"}
Nov 01 08:45:58.902078 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:58.901229    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" event={"ID":"2bd618d9-cd13-4ab2-998b-a23c6ac10545","Type":"ContainerStarted","Data":"ef491f549ae3888b4db699eb03bae9a254ac37f9aa3e7cae30abb1691c3c1682"}
Nov 01 08:45:59.001558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:59.001098    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:45:59.009932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:59.009871    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podStartSLOduration=2.009851646 podStartE2EDuration="2.009851646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:59.009339552 +0000 UTC m=+832.203734104" watchObservedRunningTime="2026-11-01 08:45:59.009851646 +0000 UTC m=+832.204246204"
Nov 01 08:45:59.800689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:45:59.800644    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:46:00.849843 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:00.849802    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:01.462111 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:01.462074    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:01.612370 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:01.612333    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:04.181421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:04.181367    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:04.448321 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:04.448232    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:07.794087 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.794053    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:46:07.806897 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.806869    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:46:07.962769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.962685    4314 generic.go:356] "Generic (PLEG): container finished" podID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerID="75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1" exitCode=0
Nov 01 08:46:07.962769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.962753    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" event={"ID":"4fd04ace-2609-40d0-9a61-82b45be1f283","Type":"ContainerDied","Data":"75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"}
Nov 01 08:46:07.963218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.963195    4314 scope.go:122] "RemoveContainer" containerID="75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"
Nov 01 08:46:07.964433 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.964412    4314 generic.go:356] "Generic (PLEG): container finished" podID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerID="41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945" exitCode=0
Nov 01 08:46:07.964512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.964488    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" event={"ID":"8fac6601-db7c-4e32-84fc-5920fa3979b8","Type":"ContainerDied","Data":"41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"}
Nov 01 08:46:07.964572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.964527    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:46:07.964627 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.964612    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:46:07.964672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.964657    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:46:07.964951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.964939    4314 scope.go:122] "RemoveContainer" containerID="41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"
Nov 01 08:46:07.966079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.966051    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:46:07.966152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.966111    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:46:07.966152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.966142    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:46:07.976555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:07.976520    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:46:08.495445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:08.495394    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:46:08.907302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:08.907212    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 08:46:08.994568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:08.994442    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:09.012210 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.012149    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" event={"ID":"4fd04ace-2609-40d0-9a61-82b45be1f283","Type":"ContainerStarted","Data":"fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4"}
Nov 01 08:46:09.014578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.014523    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" event={"ID":"8fac6601-db7c-4e32-84fc-5920fa3979b8","Type":"ContainerStarted","Data":"cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e"}
Nov 01 08:46:09.134057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.134018    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:46:09.135126 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.135098    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 08:46:09.135197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.135150    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 08:46:09.335868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.335826    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:46:09.336907 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.336883    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 08:46:09.337021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:09.336946    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 08:46:10.134390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:10.134353    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 08:46:10.134832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:10.134401    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 08:46:10.335771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:10.335735    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 08:46:10.335948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:10.335790    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 08:46:11.135020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.134987    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 08:46:11.135423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.135036    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 08:46:11.150355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.150325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:11.335976 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.335926    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 08:46:11.336168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.335981    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 08:46:11.344937 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:46:11.344901    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:46:11.649057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.649019    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:46:11.739485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.739454    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:11.887556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:11.887140    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:12.139200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:12.139142    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 08:46:12.139735 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:12.139257    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 08:46:12.335893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:12.335848    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 08:46:12.336074 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:12.335916    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 08:46:13.135141 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:13.135103    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 08:46:13.135329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:13.135170    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 08:46:13.336305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:13.336268    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 08:46:13.336740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:13.336337    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 08:46:14.134936 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:14.134894    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 08:46:14.135145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:14.134967    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 08:46:14.207950 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:14.207906    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:14.341658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:14.341629    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:46:14.342106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:14.342083    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:46:14.342980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:14.342962    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 08:46:15.139006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:15.138929    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:46:15.140028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:15.140002    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 08:46:19.115040 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:19.114965    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:19.281129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:19.281100    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:19.428466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:19.428364    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:20.249959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:20.249918    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:24.271787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:24.271745    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:24.465599 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:24.465564    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:24.843302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:24.843264    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:24.991820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:24.991787    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:25.148371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:25.148274    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:25.503053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:25.503013    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:29.186168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:29.186122    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:34.007337 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:34.007302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:38.777276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:38.777234    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:41.322461 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:46:41.322412    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:46:43.594455 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:43.594425    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:45.309944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.309912    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"]
Nov 01 08:46:45.313976 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.313953    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.397603 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.397557    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:45.409790 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.409761    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"premium-w2-343207-kserve-self-signed-certs\""
Nov 01 08:46:45.412341 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.412291    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"]
Nov 01 08:46:45.469413 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.469375    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.469641 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.469424    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnwg\" (UniqueName: \"kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.469641 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.469574    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.469747 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.469675    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.469747 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.469708    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.469822 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.469773    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.570948    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571026    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571049    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnwg\" (UniqueName: \"kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571084    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571125    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571156    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571505 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571378    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571505 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571475    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.571593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.571557    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.573218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.573188    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.573545 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.573515    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.628285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.628245    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnwg\" (UniqueName: \"kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg\") pod \"premium-w2-343207-kserve-cdb7bb78f-zrrdd\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") " pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:45.927808 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:45.927723    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:46.293813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:46.293786    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"]
Nov 01 08:46:46.295520 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:46:46.295491    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17cd82c9_b12d_4ea9_a1e5_efa69255bc4e.slice/crio-7f07e302a4047956b914c39c0acfc02f15e20adafde46ac9d4f550fef8315ff5 WatchSource:0}: Error finding container 7f07e302a4047956b914c39c0acfc02f15e20adafde46ac9d4f550fef8315ff5: Status 404 returned error can't find the container with id 7f07e302a4047956b914c39c0acfc02f15e20adafde46ac9d4f550fef8315ff5
Nov 01 08:46:47.216749 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:47.216715    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" event={"ID":"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e","Type":"ContainerStarted","Data":"8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f"}
Nov 01 08:46:47.217157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:47.216760    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" event={"ID":"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e","Type":"ContainerStarted","Data":"7f07e302a4047956b914c39c0acfc02f15e20adafde46ac9d4f550fef8315ff5"}
Nov 01 08:46:47.325466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:47.325397    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podStartSLOduration=2.3253782960000002 podStartE2EDuration="2.325378296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:46:47.325048138 +0000 UTC m=+880.519442694" watchObservedRunningTime="2026-11-01 08:46:47.325378296 +0000 UTC m=+880.519772853"
Nov 01 08:46:47.928020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:47.927986    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:48.408766 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:48.408730    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:53.168089 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:53.168049    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:55.754453 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:55.754415    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:55.928172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:55.928134    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:55.940695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:55.940668    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:56.274786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:56.274753    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:46:56.355138 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:56.355093    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:56.500872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:56.500835    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:56.649033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:56.648950    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:58.030420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:58.030385    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:46:58.353275 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:46:58.353189    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:03.177196 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:03.177156    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:04.781618 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:04.781580    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:07.970613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:07.970579    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:08.430951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.430908    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"]
Nov 01 08:47:08.434677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.434657    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.498986 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.498945    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8qd\" (UniqueName: \"kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.499175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.499104    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.499175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.499157    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.499266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.499185    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.499316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.499283    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.499384 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.499366    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.514132 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.514101    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:08.529379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.529338    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"openshift-service-ca.crt\""
Nov 01 08:47:08.529379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.529381    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"default-dockercfg-t4wrj\""
Nov 01 08:47:08.529620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.529414    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"premium-w0-751a73-kserve-self-signed-certs\""
Nov 01 08:47:08.529620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.529495    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"kube-root-ca.crt\""
Nov 01 08:47:08.531831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.531809    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"]
Nov 01 08:47:08.600891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.600851    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601082 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.600915    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8qd\" (UniqueName: \"kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601082 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601074    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601165 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601116    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601165 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601146    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601219    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601326    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601466    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.601667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.601514    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.603429 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.603406    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.603703 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.603686    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.658193 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.658156    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8qd\" (UniqueName: \"kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd\") pod \"premium-w0-751a73-kserve-85668b8d94-79lbc\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") " pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:08.747692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:08.747642    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:09.124341 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:09.124248    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"]
Nov 01 08:47:09.127519 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:47:09.127484    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369d8ecd_2471_4aab_a1c3_e099e2fb5dda.slice/crio-b60f7df135117c2b3147dca0dcd635f307db32e6be7102fe808c5bd68e333f7c WatchSource:0}: Error finding container b60f7df135117c2b3147dca0dcd635f307db32e6be7102fe808c5bd68e333f7c: Status 404 returned error can't find the container with id b60f7df135117c2b3147dca0dcd635f307db32e6be7102fe808c5bd68e333f7c
Nov 01 08:47:09.339591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:09.339522    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" event={"ID":"369d8ecd-2471-4aab-a1c3-e099e2fb5dda","Type":"ContainerStarted","Data":"aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461"}
Nov 01 08:47:09.339591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:09.339591    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" event={"ID":"369d8ecd-2471-4aab-a1c3-e099e2fb5dda","Type":"ContainerStarted","Data":"b60f7df135117c2b3147dca0dcd635f307db32e6be7102fe808c5bd68e333f7c"}
Nov 01 08:47:09.449173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:09.449061    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podStartSLOduration=1.449046374 podStartE2EDuration="1.449046374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:09.448405473 +0000 UTC m=+902.642800030" watchObservedRunningTime="2026-11-01 08:47:09.449046374 +0000 UTC m=+902.643440929"
Nov 01 08:47:09.748737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:09.748705    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:10.414145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:10.414108    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:11.323149 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:47:11.323112    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:47:12.773413 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:12.773366    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:17.633912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:17.633877    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:18.748627 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:18.748587    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:18.761497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:18.761465    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:19.408850 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:19.408822    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:47:22.088099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:22.088066    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:22.384382 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:22.384293    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:27.184393 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:27.184357    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:30.004162 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:30.004128    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:32.068829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:32.068796    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:36.886327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:36.886292    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:37.362361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.362326    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"]
Nov 01 08:47:37.366270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.366250    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.414157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.414110    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.414353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.414191    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.414353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.414224    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.414353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.414252    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.414353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.414311    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhvj\" (UniqueName: \"kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.414656 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.414379    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.456734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.456679    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:37.460779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.460744    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"kube-root-ca.crt\""
Nov 01 08:47:37.460948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.460854    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"default-dockercfg-q6wkx\""
Nov 01 08:47:37.460948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.460865    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"premium-w4-b711ae-kserve-self-signed-certs\""
Nov 01 08:47:37.460948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.460868    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"openshift-service-ca.crt\""
Nov 01 08:47:37.463425 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.463399    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"]
Nov 01 08:47:37.515709 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.515667    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.515942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.515799    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.515942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.515848    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.515942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.515879    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.515942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.515903    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.516179 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.515957    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhvj\" (UniqueName: \"kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.516272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.516251    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.517021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.516632    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.517021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.516856    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.518849 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.518797    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.519427 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.519401    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.574379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.574343    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhvj\" (UniqueName: \"kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj\") pod \"premium-w4-b711ae-kserve-f668d6bd7-tjptj\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") " pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:37.678610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:37.678504    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:38.060065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:38.060040    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"]
Nov 01 08:47:38.062106 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:47:38.062079    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54afff9d_b553_41c2_8857_8a17fa342d12.slice/crio-aee843a853f35d46c149c18fae85d3f037cebbffb691a7a03ebcf613d10b51d0 WatchSource:0}: Error finding container aee843a853f35d46c149c18fae85d3f037cebbffb691a7a03ebcf613d10b51d0: Status 404 returned error can't find the container with id aee843a853f35d46c149c18fae85d3f037cebbffb691a7a03ebcf613d10b51d0
Nov 01 08:47:38.500460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:38.500420    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" event={"ID":"54afff9d-b553-41c2-8857-8a17fa342d12","Type":"ContainerStarted","Data":"20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75"}
Nov 01 08:47:38.500460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:38.500465    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" event={"ID":"54afff9d-b553-41c2-8857-8a17fa342d12","Type":"ContainerStarted","Data":"aee843a853f35d46c149c18fae85d3f037cebbffb691a7a03ebcf613d10b51d0"}
Nov 01 08:47:38.610468 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:38.610420    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podStartSLOduration=1.610405482 podStartE2EDuration="1.610405482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:38.608973752 +0000 UTC m=+931.803368308" watchObservedRunningTime="2026-11-01 08:47:38.610405482 +0000 UTC m=+931.804800037"
Nov 01 08:47:38.679256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:38.679224    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:41.322274 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:47:41.322229    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:47:41.689274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:41.689183    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:46.490999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:46.490957    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:47.679253 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:47.679215    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:47.692053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:47.692027    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:48.568064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:48.568036    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:47:51.390249 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.390215    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:51.398166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.398130    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"]
Nov 01 08:47:51.403312 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.403290    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.500633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.500602    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:47:51.500633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.500614    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:47:51.500876 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.500643    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:47:51.500876 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.500699    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:47:51.503159 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.503138    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"]
Nov 01 08:47:51.570818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.570785    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gdp\" (UniqueName: \"kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.571018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.570824    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.571018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.570928    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.672192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.672087    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.672192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.672151    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.672434 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.672239    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gdp\" (UniqueName: \"kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.672956 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.672929    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.674582 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.674556    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:51.730136 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:51.730096    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gdp\" (UniqueName: \"kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp\") pod \"maas-api-revoke-sub-c3d6125e-4bdl6\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:52.017273 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:52.017234    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:52.387200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:52.387171    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"]
Nov 01 08:47:52.388511 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:47:52.388479    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37adea76_6c18_4dc2_95e3_8ba26cb700ce.slice/crio-2665b5abc7909a77820027e03573a84c92497fa265b8e7670a8364e1eb367678 WatchSource:0}: Error finding container 2665b5abc7909a77820027e03573a84c92497fa265b8e7670a8364e1eb367678: Status 404 returned error can't find the container with id 2665b5abc7909a77820027e03573a84c92497fa265b8e7670a8364e1eb367678
Nov 01 08:47:52.581515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:52.581486    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6" event={"ID":"37adea76-6c18-4dc2-95e3-8ba26cb700ce","Type":"ContainerStarted","Data":"2665b5abc7909a77820027e03573a84c92497fa265b8e7670a8364e1eb367678"}
Nov 01 08:47:53.587864 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:53.587831    4314 generic.go:356] "Generic (PLEG): container finished" podID="37adea76-6c18-4dc2-95e3-8ba26cb700ce" containerID="d0787ff83585bd7ea2f9d134ec964022504a5bd7bc9f8964f73188877b4b51e7" exitCode=0
Nov 01 08:47:53.588273 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:53.587920    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6" event={"ID":"37adea76-6c18-4dc2-95e3-8ba26cb700ce","Type":"ContainerDied","Data":"d0787ff83585bd7ea2f9d134ec964022504a5bd7bc9f8964f73188877b4b51e7"}
Nov 01 08:47:54.764930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.764888    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:54.910073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.909966    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca\") pod \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") "
Nov 01 08:47:54.910354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.910318    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp\" (UniqueName: \"kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp\") pod \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") "
Nov 01 08:47:54.910521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.910501    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca\") pod \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\" (UID: \"37adea76-6c18-4dc2-95e3-8ba26cb700ce\") "
Nov 01 08:47:54.912408 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.912383    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca" pod "37adea76-6c18-4dc2-95e3-8ba26cb700ce" (UID: "37adea76-6c18-4dc2-95e3-8ba26cb700ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:47:54.915520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.915494    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp" pod "37adea76-6c18-4dc2-95e3-8ba26cb700ce" (UID: "37adea76-6c18-4dc2-95e3-8ba26cb700ce"). InnerVolumeSpecName "kube-api-access-v8gdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:47:54.930451 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:54.930414    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca" pod "37adea76-6c18-4dc2-95e3-8ba26cb700ce" (UID: "37adea76-6c18-4dc2-95e3-8ba26cb700ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:47:55.013636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:55.013592    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:47:55.013896 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:55.013876    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gdp\" (UniqueName: \"kubernetes.io/projected/37adea76-6c18-4dc2-95e3-8ba26cb700ce-kube-api-access-v8gdp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:47:55.014035 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:55.014016    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37adea76-6c18-4dc2-95e3-8ba26cb700ce-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:47:55.603822 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:55.603786    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6" event={"ID":"37adea76-6c18-4dc2-95e3-8ba26cb700ce","Type":"ContainerDied","Data":"2665b5abc7909a77820027e03573a84c92497fa265b8e7670a8364e1eb367678"}
Nov 01 08:47:55.603994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:55.603830    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"
Nov 01 08:47:55.603994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:55.603859    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2665b5abc7909a77820027e03573a84c92497fa265b8e7670a8364e1eb367678"
Nov 01 08:47:56.103288 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:56.103246    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:56.252708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:56.252658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:47:57.604513 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.604477    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"]
Nov 01 08:47:57.604947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.604919    4314 cpu_manager.go:414] "containerMap: removing container" podUID="37adea76-6c18-4dc2-95e3-8ba26cb700ce" containerName="revoke-keys"
Nov 01 08:47:57.604947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.604932    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37adea76-6c18-4dc2-95e3-8ba26cb700ce" containerName="revoke-keys"
Nov 01 08:47:57.605033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.605024    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="37adea76-6c18-4dc2-95e3-8ba26cb700ce" containerName="revoke-keys"
Nov 01 08:47:57.608315 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.608297    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.656817 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.656787    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b711ae-openshift-default-dockercfg-ms66n\""
Nov 01 08:47:57.705498 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.705458    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"]
Nov 01 08:47:57.743047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743015    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743064    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwgj\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743169    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743198    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743384 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743230    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743384 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743355    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743396    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.743678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.743544    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.844469 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844434    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.844691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844496    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwgj\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.844691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844588    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.844691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844612    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.844691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844649    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.844691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844685    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844712    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844871    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.844963    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.845051    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.845077    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.845143    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.845394    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.845508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.845504    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.846833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.846812    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.847176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.847158    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.901973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.901902    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.902102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.902066    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwgj\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:57.921936 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:57.921909    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:58.341743 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.341718    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"]
Nov 01 08:47:58.343799 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:47:58.343763    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd617e251_78b9_44ee_b7a7_5996e5ede248.slice/crio-2cb5dce291f321e19c336cf0e32d3d37fa63331510f940d2bce8a47c42397fe6 WatchSource:0}: Error finding container 2cb5dce291f321e19c336cf0e32d3d37fa63331510f940d2bce8a47c42397fe6: Status 404 returned error can't find the container with id 2cb5dce291f321e19c336cf0e32d3d37fa63331510f940d2bce8a47c42397fe6
Nov 01 08:47:58.346028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.345991    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:47:58.346141 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.346074    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:47:58.346141 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.346117    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:47:58.624695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.624591    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" event={"ID":"d617e251-78b9-44ee-b7a7-5996e5ede248","Type":"ContainerStarted","Data":"78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254"}
Nov 01 08:47:58.624695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.624694    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" event={"ID":"d617e251-78b9-44ee-b7a7-5996e5ede248","Type":"ContainerStarted","Data":"2cb5dce291f321e19c336cf0e32d3d37fa63331510f940d2bce8a47c42397fe6"}
Nov 01 08:47:58.737178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.737027    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podStartSLOduration=1.737011881 podStartE2EDuration="1.737011881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:58.736470818 +0000 UTC m=+951.930865364" watchObservedRunningTime="2026-11-01 08:47:58.737011881 +0000 UTC m=+951.931406437"
Nov 01 08:47:58.927780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:58.927702    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:47:59.924655 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:59.924620    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 08:47:59.925060 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:47:59.924673    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 08:48:00.922382 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:00.922345    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 08:48:00.922716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:00.922687    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 08:48:00.996368 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:00.996332    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:01.922890 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:01.922861    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 08:48:01.923030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:01.922918    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 08:48:02.922506 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:02.922471    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 08:48:02.922931 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:02.922548    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 08:48:03.922898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:03.922860    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 08:48:03.923264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:03.922917    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 08:48:04.922383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:04.922346    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 08:48:04.922610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:04.922400    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 08:48:05.127888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:05.127855    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:05.812832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:05.812795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:05.926237 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:05.926206    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:48:05.926598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:05.926579    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:48:05.927265 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:05.927245    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 08:48:09.332420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.332381    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"]
Nov 01 08:48:09.336571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.336547    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.383182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.383140    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.383367 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.383190    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vgt\" (UniqueName: \"kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.383430 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.383366    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.386821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.386793    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:48:09.387023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.386847    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:48:09.387023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.386847    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:48:09.387023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.386928    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:48:09.390658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.390635    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"]
Nov 01 08:48:09.484601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.484558    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.484788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.484611    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vgt\" (UniqueName: \"kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.484788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.484708    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.485310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.485290    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.486893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.486875    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.541754 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.541724    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vgt\" (UniqueName: \"kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt\") pod \"maas-api-revoke-sub-fa83ef41-dqdnk\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:09.650414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:09.650333    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:10.014975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:10.014948    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"]
Nov 01 08:48:10.016814 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:48:10.016785    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9fe2fb_6be6_4790_b9a9_8e0aa7c24b80.slice/crio-cc7cd186279116e3ba1f2d00d78e98aa0067860a1bd2f42fa35686acbae861b4 WatchSource:0}: Error finding container cc7cd186279116e3ba1f2d00d78e98aa0067860a1bd2f42fa35686acbae861b4: Status 404 returned error can't find the container with id cc7cd186279116e3ba1f2d00d78e98aa0067860a1bd2f42fa35686acbae861b4
Nov 01 08:48:10.581598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:10.581497    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:10.694820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:10.694778    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk" event={"ID":"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80","Type":"ContainerStarted","Data":"430ef1acad3317c43fc35973256a60ebeeb54af7cc3db79f3e0f9a13203d45b5"}
Nov 01 08:48:10.695012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:10.694866    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk" event={"ID":"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80","Type":"ContainerStarted","Data":"cc7cd186279116e3ba1f2d00d78e98aa0067860a1bd2f42fa35686acbae861b4"}
Nov 01 08:48:10.802519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:10.802461    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk" podStartSLOduration=1.802445123 podStartE2EDuration="1.802445123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:10.801784275 +0000 UTC m=+963.996178865" watchObservedRunningTime="2026-11-01 08:48:10.802445123 +0000 UTC m=+963.996839680"
Nov 01 08:48:11.322239 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:48:11.322202    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:48:11.701912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:11.701810    4314 generic.go:356] "Generic (PLEG): container finished" podID="fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" containerID="430ef1acad3317c43fc35973256a60ebeeb54af7cc3db79f3e0f9a13203d45b5" exitCode=0
Nov 01 08:48:11.701912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:11.701889    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk" event={"ID":"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80","Type":"ContainerDied","Data":"430ef1acad3317c43fc35973256a60ebeeb54af7cc3db79f3e0f9a13203d45b5"}
Nov 01 08:48:12.850589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.850560    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:12.921518 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.921481    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt\" (UniqueName: \"kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt\") pod \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") "
Nov 01 08:48:12.921734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.921572    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca\") pod \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") "
Nov 01 08:48:12.921734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.921670    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca\") pod \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\" (UID: \"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80\") "
Nov 01 08:48:12.922024 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.921994    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca" pod "fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" (UID: "fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:48:12.923775 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.923750    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca" pod "fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" (UID: "fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:48:12.923995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:12.923962    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt" pod "fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" (UID: "fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80"). InnerVolumeSpecName "kube-api-access-58vgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:48:13.023354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:13.023309    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58vgt\" (UniqueName: \"kubernetes.io/projected/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-kube-api-access-58vgt\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:13.023354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:13.023342    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:13.023354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:13.023354    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:13.715319 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:13.715280    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk" event={"ID":"fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80","Type":"ContainerDied","Data":"cc7cd186279116e3ba1f2d00d78e98aa0067860a1bd2f42fa35686acbae861b4"}
Nov 01 08:48:13.715511 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:13.715336    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc7cd186279116e3ba1f2d00d78e98aa0067860a1bd2f42fa35686acbae861b4"
Nov 01 08:48:13.715511 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:13.715298    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"
Nov 01 08:48:15.379654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:15.379624    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:20.205108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:20.205070    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:25.019208 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:25.019177    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:25.542072 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:25.542042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:26.767914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.767881    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"]
Nov 01 08:48:26.768360 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.768346    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" containerName="revoke-keys"
Nov 01 08:48:26.768421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.768361    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" containerName="revoke-keys"
Nov 01 08:48:26.768459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.768426    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" containerName="revoke-keys"
Nov 01 08:48:26.771879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.771864    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.822397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.822345    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:48:26.868041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.868013    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:48:26.868041 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.868010    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:48:26.868249 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.868144    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:48:26.870691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.870673    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"]
Nov 01 08:48:26.878928 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.878908    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxgz\" (UniqueName: \"kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.879008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.878956    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.879073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.879051    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.979758 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.979723    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxgz\" (UniqueName: \"kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.979921 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.979813    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.979921 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.979864    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.980716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.980694    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:26.982130 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:26.982109    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:27.037582 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:27.037509    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxgz\" (UniqueName: \"kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz\") pod \"maas-api-revoke-sub-5de02b11-rdpmd\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:27.092729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:27.092692    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:27.458377 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:27.458348    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"]
Nov 01 08:48:27.459884 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:48:27.459857    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f77e973_7883_42c4_8395_e97af1530cc0.slice/crio-cae5df7c779203e4b098d100c44d206f590a4b7e7a9cb0a4aa52d804dc1a397f WatchSource:0}: Error finding container cae5df7c779203e4b098d100c44d206f590a4b7e7a9cb0a4aa52d804dc1a397f: Status 404 returned error can't find the container with id cae5df7c779203e4b098d100c44d206f590a4b7e7a9cb0a4aa52d804dc1a397f
Nov 01 08:48:27.799953 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:27.799908    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd" event={"ID":"9f77e973-7883-42c4-8395-e97af1530cc0","Type":"ContainerStarted","Data":"071d1ae600bce92c4cff5a77e21ba860330f4c5eac38a96000571666cb215ff8"}
Nov 01 08:48:27.800331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:27.799974    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd" event={"ID":"9f77e973-7883-42c4-8395-e97af1530cc0","Type":"ContainerStarted","Data":"cae5df7c779203e4b098d100c44d206f590a4b7e7a9cb0a4aa52d804dc1a397f"}
Nov 01 08:48:27.910106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:27.910044    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd" podStartSLOduration=1.910029474 podStartE2EDuration="1.910029474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:27.907124387 +0000 UTC m=+981.101518943" watchObservedRunningTime="2026-11-01 08:48:27.910029474 +0000 UTC m=+981.104424028"
Nov 01 08:48:28.678878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.678801    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"]
Nov 01 08:48:28.682441 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.682420    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.738358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.738333    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"]
Nov 01 08:48:28.800748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.800720    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.801109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.800754    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfvw7\" (UniqueName: \"kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.801109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.800847    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.806919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.806895    4314 generic.go:356] "Generic (PLEG): container finished" podID="9f77e973-7883-42c4-8395-e97af1530cc0" containerID="071d1ae600bce92c4cff5a77e21ba860330f4c5eac38a96000571666cb215ff8" exitCode=0
Nov 01 08:48:28.807032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.806966    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd" event={"ID":"9f77e973-7883-42c4-8395-e97af1530cc0","Type":"ContainerDied","Data":"071d1ae600bce92c4cff5a77e21ba860330f4c5eac38a96000571666cb215ff8"}
Nov 01 08:48:28.901827 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.901800    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.901980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.901834    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfvw7\" (UniqueName: \"kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.901980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.901874    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.902583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.902565    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.904072 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.904045    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.959233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.959173    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfvw7\" (UniqueName: \"kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7\") pod \"maas-api-revoke-sub-4f536057-j7qsd\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:28.996127 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:28.996098    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:29.360021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:29.359994    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"]
Nov 01 08:48:29.361359 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:48:29.361327    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5789434_7044_43fe_be5f_4a2a2784300d.slice/crio-f555349c096ad5f72c42395812e1f39e59cc0b4c4c6dc4403ddeff91edf65487 WatchSource:0}: Error finding container f555349c096ad5f72c42395812e1f39e59cc0b4c4c6dc4403ddeff91edf65487: Status 404 returned error can't find the container with id f555349c096ad5f72c42395812e1f39e59cc0b4c4c6dc4403ddeff91edf65487
Nov 01 08:48:29.814437 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:29.814402    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd" event={"ID":"a5789434-7044-43fe-be5f-4a2a2784300d","Type":"ContainerStarted","Data":"c89d0e38a81dac14f215c813512edd07ff354e7008f131d3df9506d226f9f30d"}
Nov 01 08:48:29.814872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:29.814455    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd" event={"ID":"a5789434-7044-43fe-be5f-4a2a2784300d","Type":"ContainerStarted","Data":"f555349c096ad5f72c42395812e1f39e59cc0b4c4c6dc4403ddeff91edf65487"}
Nov 01 08:48:29.892851 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:29.892817    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:29.925155 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:29.925096    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd" podStartSLOduration=1.925081722 podStartE2EDuration="1.925081722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:29.922673323 +0000 UTC m=+983.117072627" watchObservedRunningTime="2026-11-01 08:48:29.925081722 +0000 UTC m=+983.119476277"
Nov 01 08:48:29.944819 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:29.944792    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:30.011604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.011564    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz\" (UniqueName: \"kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz\") pod \"9f77e973-7883-42c4-8395-e97af1530cc0\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") "
Nov 01 08:48:30.011766 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.011621    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca\") pod \"9f77e973-7883-42c4-8395-e97af1530cc0\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") "
Nov 01 08:48:30.011766 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.011658    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca\") pod \"9f77e973-7883-42c4-8395-e97af1530cc0\" (UID: \"9f77e973-7883-42c4-8395-e97af1530cc0\") "
Nov 01 08:48:30.012055 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.012034    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca" pod "9f77e973-7883-42c4-8395-e97af1530cc0" (UID: "9f77e973-7883-42c4-8395-e97af1530cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:48:30.013685 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.013660    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca" pod "9f77e973-7883-42c4-8395-e97af1530cc0" (UID: "9f77e973-7883-42c4-8395-e97af1530cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:48:30.013817 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.013776    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz" pod "9f77e973-7883-42c4-8395-e97af1530cc0" (UID: "9f77e973-7883-42c4-8395-e97af1530cc0"). InnerVolumeSpecName "kube-api-access-qpxgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:48:30.112557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.112500    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxgz\" (UniqueName: \"kubernetes.io/projected/9f77e973-7883-42c4-8395-e97af1530cc0-kube-api-access-qpxgz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:30.112557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.112560    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:30.112757 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.112577    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77e973-7883-42c4-8395-e97af1530cc0-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:30.821602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.821566    4314 generic.go:356] "Generic (PLEG): container finished" podID="a5789434-7044-43fe-be5f-4a2a2784300d" containerID="c89d0e38a81dac14f215c813512edd07ff354e7008f131d3df9506d226f9f30d" exitCode=0
Nov 01 08:48:30.822025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.821659    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd" event={"ID":"a5789434-7044-43fe-be5f-4a2a2784300d","Type":"ContainerDied","Data":"c89d0e38a81dac14f215c813512edd07ff354e7008f131d3df9506d226f9f30d"}
Nov 01 08:48:30.823013 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.822983    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd" event={"ID":"9f77e973-7883-42c4-8395-e97af1530cc0","Type":"ContainerDied","Data":"cae5df7c779203e4b098d100c44d206f590a4b7e7a9cb0a4aa52d804dc1a397f"}
Nov 01 08:48:30.823110 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.823034    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae5df7c779203e4b098d100c44d206f590a4b7e7a9cb0a4aa52d804dc1a397f"
Nov 01 08:48:30.823110 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:30.822999    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"
Nov 01 08:48:31.968303 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:31.968271    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:32.028608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.028574    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca\") pod \"a5789434-7044-43fe-be5f-4a2a2784300d\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") "
Nov 01 08:48:32.028813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.028635    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca\") pod \"a5789434-7044-43fe-be5f-4a2a2784300d\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") "
Nov 01 08:48:32.028813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.028711    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7\" (UniqueName: \"kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7\") pod \"a5789434-7044-43fe-be5f-4a2a2784300d\" (UID: \"a5789434-7044-43fe-be5f-4a2a2784300d\") "
Nov 01 08:48:32.028972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.028949    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca" pod "a5789434-7044-43fe-be5f-4a2a2784300d" (UID: "a5789434-7044-43fe-be5f-4a2a2784300d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:48:32.030813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.030792    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7" pod "a5789434-7044-43fe-be5f-4a2a2784300d" (UID: "a5789434-7044-43fe-be5f-4a2a2784300d"). InnerVolumeSpecName "kube-api-access-bfvw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:48:32.030954 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.030933    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca" pod "a5789434-7044-43fe-be5f-4a2a2784300d" (UID: "a5789434-7044-43fe-be5f-4a2a2784300d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:48:32.130090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.130012    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:32.130090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.130039    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5789434-7044-43fe-be5f-4a2a2784300d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:32.130090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.130049    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfvw7\" (UniqueName: \"kubernetes.io/projected/a5789434-7044-43fe-be5f-4a2a2784300d-kube-api-access-bfvw7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:32.835635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.835607    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"
Nov 01 08:48:32.835635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.835627    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd" event={"ID":"a5789434-7044-43fe-be5f-4a2a2784300d","Type":"ContainerDied","Data":"f555349c096ad5f72c42395812e1f39e59cc0b4c4c6dc4403ddeff91edf65487"}
Nov 01 08:48:32.835850 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:32.835677    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f555349c096ad5f72c42395812e1f39e59cc0b4c4c6dc4403ddeff91edf65487"
Nov 01 08:48:34.677185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:34.677154    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:39.490318 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:39.490287    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:41.321923 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:48:41.321874    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:48:42.552058 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552022    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"]
Nov 01 08:48:42.552495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552482    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9f77e973-7883-42c4-8395-e97af1530cc0" containerName="revoke-keys"
Nov 01 08:48:42.552563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552496    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f77e973-7883-42c4-8395-e97af1530cc0" containerName="revoke-keys"
Nov 01 08:48:42.552563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552521    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a5789434-7044-43fe-be5f-4a2a2784300d" containerName="revoke-keys"
Nov 01 08:48:42.552563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552526    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5789434-7044-43fe-be5f-4a2a2784300d" containerName="revoke-keys"
Nov 01 08:48:42.552668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552608    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5789434-7044-43fe-be5f-4a2a2784300d" containerName="revoke-keys"
Nov 01 08:48:42.552668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.552622    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f77e973-7883-42c4-8395-e97af1530cc0" containerName="revoke-keys"
Nov 01 08:48:42.556283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.556264    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.609993 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.609960    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"]
Nov 01 08:48:42.634076 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.634042    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8gs\" (UniqueName: \"kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.634260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.634101    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.634260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.634167    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.652470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.652445    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:48:42.652470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.652458    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:48:42.652470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.652473    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:48:42.652711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.652472    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:48:42.735398 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.735366    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8gs\" (UniqueName: \"kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.735604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.735429    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.735604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.735459    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.736160 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.736139    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.737888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.737868    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.792155 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.792124    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8gs\" (UniqueName: \"kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs\") pod \"maas-api-revoke-sub-996da403-lfcdd\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:42.872097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:42.872026    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:43.245659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:43.245626    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"]
Nov 01 08:48:43.897901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:43.897803    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd" event={"ID":"83eaa36f-a4a4-4571-89e6-0873c9a602eb","Type":"ContainerStarted","Data":"f5665af640346f5ad879b1fae0d268ffdee07550ff273d87f6f47c185b531e84"}
Nov 01 08:48:43.897901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:43.897869    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd" event={"ID":"83eaa36f-a4a4-4571-89e6-0873c9a602eb","Type":"ContainerStarted","Data":"7e30d34f503210b53e6116b2121a0657c4e9d8aebdefa982f043606bce3749ba"}
Nov 01 08:48:44.007889 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:44.007836    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd" podStartSLOduration=2.007821576 podStartE2EDuration="2.007821576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:44.007073914 +0000 UTC m=+997.201468467" watchObservedRunningTime="2026-11-01 08:48:44.007821576 +0000 UTC m=+997.202216130"
Nov 01 08:48:44.302997 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:44.302963    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:44.904371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:44.904333    4314 generic.go:356] "Generic (PLEG): container finished" podID="83eaa36f-a4a4-4571-89e6-0873c9a602eb" containerID="f5665af640346f5ad879b1fae0d268ffdee07550ff273d87f6f47c185b531e84" exitCode=0
Nov 01 08:48:44.904788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:44.904416    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd" event={"ID":"83eaa36f-a4a4-4571-89e6-0873c9a602eb","Type":"ContainerDied","Data":"f5665af640346f5ad879b1fae0d268ffdee07550ff273d87f6f47c185b531e84"}
Nov 01 08:48:46.052269 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.052244    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:46.177006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.176915    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca\") pod \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") "
Nov 01 08:48:46.177006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.176969    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs\" (UniqueName: \"kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs\") pod \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") "
Nov 01 08:48:46.177256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.177094    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca\") pod \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\" (UID: \"83eaa36f-a4a4-4571-89e6-0873c9a602eb\") "
Nov 01 08:48:46.177355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.177329    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca" pod "83eaa36f-a4a4-4571-89e6-0873c9a602eb" (UID: "83eaa36f-a4a4-4571-89e6-0873c9a602eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:48:46.177556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.177515    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:46.179859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.179834    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs" pod "83eaa36f-a4a4-4571-89e6-0873c9a602eb" (UID: "83eaa36f-a4a4-4571-89e6-0873c9a602eb"). InnerVolumeSpecName "kube-api-access-gt8gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:48:46.180065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.180046    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca" pod "83eaa36f-a4a4-4571-89e6-0873c9a602eb" (UID: "83eaa36f-a4a4-4571-89e6-0873c9a602eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:48:46.279137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.279096    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eaa36f-a4a4-4571-89e6-0873c9a602eb-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:46.279137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.279141    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8gs\" (UniqueName: \"kubernetes.io/projected/83eaa36f-a4a4-4571-89e6-0873c9a602eb-kube-api-access-gt8gs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:46.918256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.918222    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"
Nov 01 08:48:46.918256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.918248    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd" event={"ID":"83eaa36f-a4a4-4571-89e6-0873c9a602eb","Type":"ContainerDied","Data":"7e30d34f503210b53e6116b2121a0657c4e9d8aebdefa982f043606bce3749ba"}
Nov 01 08:48:46.918464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:46.918295    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e30d34f503210b53e6116b2121a0657c4e9d8aebdefa982f043606bce3749ba"
Nov 01 08:48:48.978977 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:48.978947    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"]
Nov 01 08:48:48.979408 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:48.979394    4314 cpu_manager.go:414] "containerMap: removing container" podUID="83eaa36f-a4a4-4571-89e6-0873c9a602eb" containerName="revoke-keys"
Nov 01 08:48:48.979452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:48.979409    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83eaa36f-a4a4-4571-89e6-0873c9a602eb" containerName="revoke-keys"
Nov 01 08:48:48.979516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:48.979506    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="83eaa36f-a4a4-4571-89e6-0873c9a602eb" containerName="revoke-keys"
Nov 01 08:48:48.984180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:48.984163    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.036947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.036912    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"]
Nov 01 08:48:49.079340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.079299    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:48:49.079340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.079323    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:48:49.079340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.079316    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:48:49.079588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.079301    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:48:49.107966 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.107941    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.108077 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.107986    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.108077 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.108003    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.209209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.209169    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.209403 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.209231    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.209403 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.209259    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.209900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.209874    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:49.210129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.210110    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.211711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.211685    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.267021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.266984    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m\") pod \"maas-api-revoke-sub-c2769b8e-vwrns\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.297749 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.297723    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:49.664806 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.664780    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"]
Nov 01 08:48:49.665873 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:48:49.665841    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode198ca46_d32e_4606_99fc_45e1070d5364.slice/crio-f3f0159de57ba906020f52454847eec0f860866abba92c621c06ea2876d3f309 WatchSource:0}: Error finding container f3f0159de57ba906020f52454847eec0f860866abba92c621c06ea2876d3f309: Status 404 returned error can't find the container with id f3f0159de57ba906020f52454847eec0f860866abba92c621c06ea2876d3f309
Nov 01 08:48:49.942260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:49.942179    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns" event={"ID":"e198ca46-d32e-4606-99fc-45e1070d5364","Type":"ContainerStarted","Data":"f3f0159de57ba906020f52454847eec0f860866abba92c621c06ea2876d3f309"}
Nov 01 08:48:50.949292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:50.949255    4314 generic.go:356] "Generic (PLEG): container finished" podID="e198ca46-d32e-4606-99fc-45e1070d5364" containerID="c012abea063f50969fb3737bb870efd21731ede76ffa0cec4e3706128980d298" exitCode=0
Nov 01 08:48:50.949716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:50.949294    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns" event={"ID":"e198ca46-d32e-4606-99fc-45e1070d5364","Type":"ContainerDied","Data":"c012abea063f50969fb3737bb870efd21731ede76ffa0cec4e3706128980d298"}
Nov 01 08:48:52.083179 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.083156    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:52.239682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.239657    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca\") pod \"e198ca46-d32e-4606-99fc-45e1070d5364\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") "
Nov 01 08:48:52.239828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.239690    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m\") pod \"e198ca46-d32e-4606-99fc-45e1070d5364\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") "
Nov 01 08:48:52.239828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.239726    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca\") pod \"e198ca46-d32e-4606-99fc-45e1070d5364\" (UID: \"e198ca46-d32e-4606-99fc-45e1070d5364\") "
Nov 01 08:48:52.240102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.240078    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca" pod "e198ca46-d32e-4606-99fc-45e1070d5364" (UID: "e198ca46-d32e-4606-99fc-45e1070d5364"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:48:52.241825 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.241806    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca" pod "e198ca46-d32e-4606-99fc-45e1070d5364" (UID: "e198ca46-d32e-4606-99fc-45e1070d5364"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:48:52.241884 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.241861    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m" pod "e198ca46-d32e-4606-99fc-45e1070d5364" (UID: "e198ca46-d32e-4606-99fc-45e1070d5364"). InnerVolumeSpecName "kube-api-access-tns6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:48:52.341283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.341259    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:52.341283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.341281    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/e198ca46-d32e-4606-99fc-45e1070d5364-kube-api-access-tns6m\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:52.341430 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.341292    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198ca46-d32e-4606-99fc-45e1070d5364-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:48:52.961331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.961305    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"
Nov 01 08:48:52.961331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.961323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns" event={"ID":"e198ca46-d32e-4606-99fc-45e1070d5364","Type":"ContainerDied","Data":"f3f0159de57ba906020f52454847eec0f860866abba92c621c06ea2876d3f309"}
Nov 01 08:48:52.961520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:52.961369    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3f0159de57ba906020f52454847eec0f860866abba92c621c06ea2876d3f309"
Nov 01 08:48:53.993615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:53.993586    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:58.816403 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:58.816364    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:48:59.775647 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.775616    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"]
Nov 01 08:48:59.776051 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.776039    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e198ca46-d32e-4606-99fc-45e1070d5364" containerName="revoke-keys"
Nov 01 08:48:59.776102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.776051    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e198ca46-d32e-4606-99fc-45e1070d5364" containerName="revoke-keys"
Nov 01 08:48:59.776161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.776152    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e198ca46-d32e-4606-99fc-45e1070d5364" containerName="revoke-keys"
Nov 01 08:48:59.779518 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.779501    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:48:59.875865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.875832    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:48:59.875865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.875831    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:48:59.876377 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.875847    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:48:59.876554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.876521    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:48:59.878803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.878779    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"]
Nov 01 08:48:59.916336 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.916315    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:48:59.916461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.916345    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xj6f\" (UniqueName: \"kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:48:59.916569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:48:59.916522    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.017452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.017420    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.017631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.017466    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xj6f\" (UniqueName: \"kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.017756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.017728    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.018332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.018313    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.019774 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.019751    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.074626 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.074550    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xj6f\" (UniqueName: \"kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f\") pod \"maas-api-revoke-sub-49527f3f-rc5bn\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.093108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.093077    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:00.462297 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:49:00.462256    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21033d22_a443_489d_a1df_2336b8091dfb.slice/crio-a17e5b3722387b6f1981730d45abc8a649623f3f138feea9a73d1eef7aac6deb WatchSource:0}: Error finding container a17e5b3722387b6f1981730d45abc8a649623f3f138feea9a73d1eef7aac6deb: Status 404 returned error can't find the container with id a17e5b3722387b6f1981730d45abc8a649623f3f138feea9a73d1eef7aac6deb
Nov 01 08:49:00.483625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:00.483600    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"]
Nov 01 08:49:01.017943 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:01.017855    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn" event={"ID":"21033d22-a443-489d-a1df-2336b8091dfb","Type":"ContainerStarted","Data":"6f1d2cb837231ea3f630d8cfe16483ce77ccf5b82578c41ea29c5a97a4c340b4"}
Nov 01 08:49:01.018308 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:01.017946    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn" event={"ID":"21033d22-a443-489d-a1df-2336b8091dfb","Type":"ContainerStarted","Data":"a17e5b3722387b6f1981730d45abc8a649623f3f138feea9a73d1eef7aac6deb"}
Nov 01 08:49:01.132767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:01.132703    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn" podStartSLOduration=2.132688241 podStartE2EDuration="2.132688241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:01.131901894 +0000 UTC m=+1014.326296440" watchObservedRunningTime="2026-11-01 08:49:01.132688241 +0000 UTC m=+1014.327082797"
Nov 01 08:49:01.973224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:01.973188    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"]
Nov 01 08:49:01.977086 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:01.977063    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.025211 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.025178    4314 generic.go:356] "Generic (PLEG): container finished" podID="21033d22-a443-489d-a1df-2336b8091dfb" containerID="6f1d2cb837231ea3f630d8cfe16483ce77ccf5b82578c41ea29c5a97a4c340b4" exitCode=0
Nov 01 08:49:02.025601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.025264    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn" event={"ID":"21033d22-a443-489d-a1df-2336b8091dfb","Type":"ContainerDied","Data":"6f1d2cb837231ea3f630d8cfe16483ce77ccf5b82578c41ea29c5a97a4c340b4"}
Nov 01 08:49:02.025825 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.025810    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfdd13-openshift-default-dockercfg-jf87f\""
Nov 01 08:49:02.075298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.075270    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"]
Nov 01 08:49:02.142730 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.142692    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8p7v\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.142878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.142749    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.142878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.142818    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.142878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.142835    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.142986 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.142899    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.142986 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.142945    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.143049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.143007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.143080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.143049    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.143134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.143122    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244383    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244660 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244436    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244660 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244557    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244660 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244592    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244728    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244775    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.244855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244822    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244918    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.244922    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245140 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.245051    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8p7v\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.245157    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245387 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.245358    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.245476    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.245644 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.245624    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.246814 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.246793    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.247236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.247011    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.301710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.301681    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.301941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.301922    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8p7v\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:02.590608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:02.590500    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:03.071256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.071224    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"]
Nov 01 08:49:03.072201 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:49:03.072166    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6432463_4ec0_4323_98e8_4ea25e21c415.slice/crio-8af268935714bbd3e98f7d2367dfca9f72d185d4bf198b6676aba372231f5623 WatchSource:0}: Error finding container 8af268935714bbd3e98f7d2367dfca9f72d185d4bf198b6676aba372231f5623: Status 404 returned error can't find the container with id 8af268935714bbd3e98f7d2367dfca9f72d185d4bf198b6676aba372231f5623
Nov 01 08:49:03.074639 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.074603    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:03.074723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.074668    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:03.074723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.074701    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:03.157322 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.157304    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:03.256601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.256406    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca\") pod \"21033d22-a443-489d-a1df-2336b8091dfb\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") "
Nov 01 08:49:03.256601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.256585    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f\" (UniqueName: \"kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f\") pod \"21033d22-a443-489d-a1df-2336b8091dfb\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") "
Nov 01 08:49:03.256785 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.256662    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca\") pod \"21033d22-a443-489d-a1df-2336b8091dfb\" (UID: \"21033d22-a443-489d-a1df-2336b8091dfb\") "
Nov 01 08:49:03.256845 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.256799    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca" pod "21033d22-a443-489d-a1df-2336b8091dfb" (UID: "21033d22-a443-489d-a1df-2336b8091dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:49:03.257120 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.257095    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:49:03.258946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.258925    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f" pod "21033d22-a443-489d-a1df-2336b8091dfb" (UID: "21033d22-a443-489d-a1df-2336b8091dfb"). InnerVolumeSpecName "kube-api-access-4xj6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:49:03.259060 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.259041    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca" pod "21033d22-a443-489d-a1df-2336b8091dfb" (UID: "21033d22-a443-489d-a1df-2336b8091dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:49:03.358658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.358573    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21033d22-a443-489d-a1df-2336b8091dfb-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:49:03.358658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.358613    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xj6f\" (UniqueName: \"kubernetes.io/projected/21033d22-a443-489d-a1df-2336b8091dfb-kube-api-access-4xj6f\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:49:03.611966 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:03.611885    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:04.037836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.037798    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" event={"ID":"e6432463-4ec0-4323-98e8-4ea25e21c415","Type":"ContainerStarted","Data":"bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad"}
Nov 01 08:49:04.038049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.037862    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" event={"ID":"e6432463-4ec0-4323-98e8-4ea25e21c415","Type":"ContainerStarted","Data":"8af268935714bbd3e98f7d2367dfca9f72d185d4bf198b6676aba372231f5623"}
Nov 01 08:49:04.039292 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.039262    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn" event={"ID":"21033d22-a443-489d-a1df-2336b8091dfb","Type":"ContainerDied","Data":"a17e5b3722387b6f1981730d45abc8a649623f3f138feea9a73d1eef7aac6deb"}
Nov 01 08:49:04.039403 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.039307    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17e5b3722387b6f1981730d45abc8a649623f3f138feea9a73d1eef7aac6deb"
Nov 01 08:49:04.039403 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.039276    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"
Nov 01 08:49:04.216018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.215966    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podStartSLOduration=3.215951918 podStartE2EDuration="3.215951918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:04.195841743 +0000 UTC m=+1017.390236335" watchObservedRunningTime="2026-11-01 08:49:04.215951918 +0000 UTC m=+1017.410346473"
Nov 01 08:49:04.591011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.590977    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:04.592793 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.592767    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:04.592923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:04.592814    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 08:49:05.483958 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:05.483923    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:05.591109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:05.591077    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:05.591293 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:05.591138    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 08:49:06.591697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:06.591663    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:06.592144 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:06.591721    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 08:49:07.591247 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:07.591203    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:07.591401 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:07.591269    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 08:49:08.405002 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:08.404965    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:08.591595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:08.591560    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:08.591765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:08.591614    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 08:49:09.987943 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:09.987907    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:09.989033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:09.988970    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:09.991620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:09.991597    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 08:49:11.323006 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:49:11.322971    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:49:13.194060 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:13.194030    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:15.646713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:15.646676    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:15.793211 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:15.793180    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:15.940829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:15.940743    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:17.982724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:17.982684    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:20.073643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.073561    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"]
Nov 01 08:49:20.074040 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.074030    4314 cpu_manager.go:414] "containerMap: removing container" podUID="21033d22-a443-489d-a1df-2336b8091dfb" containerName="revoke-keys"
Nov 01 08:49:20.074088 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.074041    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21033d22-a443-489d-a1df-2336b8091dfb" containerName="revoke-keys"
Nov 01 08:49:20.074138 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.074127    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="21033d22-a443-489d-a1df-2336b8091dfb" containerName="revoke-keys"
Nov 01 08:49:20.079019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.078998    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.131872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.131840    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"]
Nov 01 08:49:20.143488 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.143455    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.143692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.143611    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.143692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.143656    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdct\" (UniqueName: \"kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.175150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.175117    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:49:20.175150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.175137    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:49:20.175150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.175117    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:49:20.176618 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.176602    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:49:20.245133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.245101    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.245420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.245166    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdct\" (UniqueName: \"kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.245420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.245240    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.246096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.246071    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.247803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.247780    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.302236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.302201    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdct\" (UniqueName: \"kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct\") pod \"maas-api-revoke-sub-a37ca150-q528f\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.392511 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.392428    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:20.763631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:20.763600    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"]
Nov 01 08:49:20.764859 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:49:20.764831    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cee309_adc9_4520_ae57_41462a339e27.slice/crio-68da32b97ae3d70b796c75b512a5bb12e6f17ef3c669e3cb7ccd8d63a66ee958 WatchSource:0}: Error finding container 68da32b97ae3d70b796c75b512a5bb12e6f17ef3c669e3cb7ccd8d63a66ee958: Status 404 returned error can't find the container with id 68da32b97ae3d70b796c75b512a5bb12e6f17ef3c669e3cb7ccd8d63a66ee958
Nov 01 08:49:21.080135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.080102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:21.152905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.152866    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f" event={"ID":"06cee309-adc9-4520-ae57-41462a339e27","Type":"ContainerStarted","Data":"4d7cbc4a4c9cebab4394295192cb29ff3bf631d6c7188bc309357c9e923eff7c"}
Nov 01 08:49:21.153114 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.152934    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f" event={"ID":"06cee309-adc9-4520-ae57-41462a339e27","Type":"ContainerStarted","Data":"68da32b97ae3d70b796c75b512a5bb12e6f17ef3c669e3cb7ccd8d63a66ee958"}
Nov 01 08:49:21.590798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.590728    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f" podStartSLOduration=1.590706077 podStartE2EDuration="1.590706077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:21.285268545 +0000 UTC m=+1034.479663102" watchObservedRunningTime="2026-11-01 08:49:21.590706077 +0000 UTC m=+1034.785100647"
Nov 01 08:49:21.592922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.592884    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"]
Nov 01 08:49:21.598010 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.597990    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.649612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.649580    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"]
Nov 01 08:49:21.653601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.653571    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-51b2b41abcdb\""
Nov 01 08:49:21.653757 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.653722    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f77f55536278\""
Nov 01 08:49:21.654079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.653985    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 08:49:21.654079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.654004    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 08:49:21.654079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.654017    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 08:49:21.654079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.654045    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 08:49:21.654346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.654258    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d7bf9\""
Nov 01 08:49:21.663167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.663093    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.663250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.663184    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvjm\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.663693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.663248    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.663693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.663432    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.764518 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.764486    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.764746 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.764578    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.764746 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.764626    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvjm\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.764746 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.764665    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.765579 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.765274    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.765700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.765675    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.767257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.767223    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.822780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.822731    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvjm\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm\") pod \"payload-processing-external-model-555b8b678f-ptnh6\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") " pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:21.919683 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:21.919576    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:22.160874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:22.160830    4314 generic.go:356] "Generic (PLEG): container finished" podID="06cee309-adc9-4520-ae57-41462a339e27" containerID="4d7cbc4a4c9cebab4394295192cb29ff3bf631d6c7188bc309357c9e923eff7c" exitCode=0
Nov 01 08:49:22.161315 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:22.160911    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f" event={"ID":"06cee309-adc9-4520-ae57-41462a339e27","Type":"ContainerDied","Data":"4d7cbc4a4c9cebab4394295192cb29ff3bf631d6c7188bc309357c9e923eff7c"}
Nov 01 08:49:22.377343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:22.377313    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"]
Nov 01 08:49:22.379202 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:49:22.379156    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e52afd4_7270_4dd9_bae3_8df8e5938bd0.slice/crio-de307ceeb421fcab596b13b9ecbb302fabcf44661be9cabc57ac645c155bce0b WatchSource:0}: Error finding container de307ceeb421fcab596b13b9ecbb302fabcf44661be9cabc57ac645c155bce0b: Status 404 returned error can't find the container with id de307ceeb421fcab596b13b9ecbb302fabcf44661be9cabc57ac645c155bce0b
Nov 01 08:49:22.836259 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:22.836182    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:23.050644 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.050610    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"]
Nov 01 08:49:23.054375 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.054359    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.152094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.152020    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"]
Nov 01 08:49:23.178571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.174850    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6" event={"ID":"0e52afd4-7270-4dd9-bae3-8df8e5938bd0","Type":"ContainerStarted","Data":"12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35"}
Nov 01 08:49:23.178571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.174929    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6" event={"ID":"0e52afd4-7270-4dd9-bae3-8df8e5938bd0","Type":"ContainerStarted","Data":"de307ceeb421fcab596b13b9ecbb302fabcf44661be9cabc57ac645c155bce0b"}
Nov 01 08:49:23.178571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.175728    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:23.190859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.190814    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.190870    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.190917    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.190990    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.191021    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191267 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.191089    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191267 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.191124    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191267 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.191153    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppx97\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.191267 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.191181    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.302556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.300173    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6" podStartSLOduration=2.30014948 podStartE2EDuration="2.30014948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:23.286961946 +0000 UTC m=+1036.481356526" watchObservedRunningTime="2026-11-01 08:49:23.30014948 +0000 UTC m=+1036.494544036"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.327818    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.327929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.327966    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.327997    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppx97\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.328029    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.328143    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.328175    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.328263    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.328556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.328378    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.330573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.329208    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.330573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.329965    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.331220 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.331196    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.331689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.331653    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.331889 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.331862    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.334563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.334400    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.338243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.338217    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.389961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.389900    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppx97\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.395651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.395591    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:23.432979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.432904    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:23.529571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.529519    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct\" (UniqueName: \"kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct\") pod \"06cee309-adc9-4520-ae57-41462a339e27\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") "
Nov 01 08:49:23.529759 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.529596    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca\") pod \"06cee309-adc9-4520-ae57-41462a339e27\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") "
Nov 01 08:49:23.529759 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.529662    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca\") pod \"06cee309-adc9-4520-ae57-41462a339e27\" (UID: \"06cee309-adc9-4520-ae57-41462a339e27\") "
Nov 01 08:49:23.529929 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.529901    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca" pod "06cee309-adc9-4520-ae57-41462a339e27" (UID: "06cee309-adc9-4520-ae57-41462a339e27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:49:23.530023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.530006    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cee309-adc9-4520-ae57-41462a339e27-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:49:23.531672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.531643    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca" pod "06cee309-adc9-4520-ae57-41462a339e27" (UID: "06cee309-adc9-4520-ae57-41462a339e27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:49:23.531672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.531653    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct" pod "06cee309-adc9-4520-ae57-41462a339e27" (UID: "06cee309-adc9-4520-ae57-41462a339e27"). InnerVolumeSpecName "kube-api-access-jvdct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:49:23.631026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.630990    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cee309-adc9-4520-ae57-41462a339e27-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:49:23.631026 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.631026    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvdct\" (UniqueName: \"kubernetes.io/projected/06cee309-adc9-4520-ae57-41462a339e27-kube-api-access-jvdct\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:49:23.672180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:23.672128    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:24.121309 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.117665    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"]
Nov 01 08:49:24.124414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.124366    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:24.124621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.124464    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:24.124621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.124509    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:24.193588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.190945    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"
Nov 01 08:49:24.195372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.188683    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" event={"ID":"468b79bd-a23b-4047-b56e-a1e5c515bcc0","Type":"ContainerStarted","Data":"8038e0196da7f96a0ff161a4b1826b9a7a9c78089bcde21b9d6a025498c08984"}
Nov 01 08:49:24.195503 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.195413    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f" event={"ID":"06cee309-adc9-4520-ae57-41462a339e27","Type":"ContainerDied","Data":"68da32b97ae3d70b796c75b512a5bb12e6f17ef3c669e3cb7ccd8d63a66ee958"}
Nov 01 08:49:24.195503 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:24.195450    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68da32b97ae3d70b796c75b512a5bb12e6f17ef3c669e3cb7ccd8d63a66ee958"
Nov 01 08:49:25.227333 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:25.227288    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" event={"ID":"468b79bd-a23b-4047-b56e-a1e5c515bcc0","Type":"ContainerStarted","Data":"407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7"}
Nov 01 08:49:25.682557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:25.682047    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:25.683578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:25.683138    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:49:25.683578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:25.683191    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 08:49:26.683126 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:26.683086    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:49:26.683659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:26.683147    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 08:49:27.683559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:27.682936    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:49:27.683559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:27.683001    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 08:49:27.916722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:27.916680    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:28.677676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:28.677633    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:49:28.677970 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:28.677940    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 08:49:29.682562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:29.675767    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:49:29.682562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:29.675831    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 08:49:30.673633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:30.673597    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:49:30.673823 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:30.673652    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 08:49:31.677135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:31.677094    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:31.677563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:31.677377    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:31.678116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:31.678098    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:49:31.839471 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:31.839417    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podStartSLOduration=8.839400943 podStartE2EDuration="8.839400943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:25.431233109 +0000 UTC m=+1038.625627662" watchObservedRunningTime="2026-11-01 08:49:31.839400943 +0000 UTC m=+1045.033795535"
Nov 01 08:49:32.740731 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:32.740693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:34.198164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:34.198131    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:49:34.970895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:34.970861    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:37.578343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:37.578303    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:41.321612 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:49:41.321576    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:49:42.578221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:42.578163    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:47.576505 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:47.576466    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:52.475682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:52.475640    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:53.972127 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:53.972098    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"]
Nov 01 08:49:53.972596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:53.972570    4314 cpu_manager.go:414] "containerMap: removing container" podUID="06cee309-adc9-4520-ae57-41462a339e27" containerName="revoke-keys"
Nov 01 08:49:53.972596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:53.972582    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cee309-adc9-4520-ae57-41462a339e27" containerName="revoke-keys"
Nov 01 08:49:53.972712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:53.972659    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cee309-adc9-4520-ae57-41462a339e27" containerName="revoke-keys"
Nov 01 08:49:53.975654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:53.975636    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.017954 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.017931    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"]
Nov 01 08:49:54.021366 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.021348    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101140    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101188    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101234    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101258    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101285    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101355 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101336    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101436    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z6t\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101464    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101504    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101553    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101605    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101648    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101679    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101703    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101749    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101819    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.101926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101894    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqc9d\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.101926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.101919    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.103981 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.103959    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"]
Nov 01 08:49:54.178671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.178639    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"]
Nov 01 08:49:54.203160 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203126    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203160 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203159    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203179    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203205    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203240    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203311    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z6t\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203338    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203368    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203397    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203445    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203475    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203506    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203548    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203581    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203648    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203701    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqc9d\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.203716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203717    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.204027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203762    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.204027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.203837    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.204194 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.204173    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.204361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.204223    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.204361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.204261    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.204512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.204456    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.204791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.204770    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.205141 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.205110    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.205313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.205290    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.205464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.205442    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.205709 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.205690    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.206291 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.206269    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.206678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.206657    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.207381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.207352    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.207479 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.207461    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.260004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.259983    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.260313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.260290    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z6t\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.260389 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.260292    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tfphq\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.260445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.260406    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqc9d\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d\") pod \"maas-default-gateway-openshift-default-746ff8f49f-sdwlj\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.289370 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.289344    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:54.333857 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.333829    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:54.704828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.704806    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"]
Nov 01 08:49:54.705869 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:49:54.705844    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf477d9c_9a2f_4618_998a_150e6baa9147.slice/crio-3fe9b84c2c494c46a0d9f69fa8d431365d83287e0bef4f201c5f3e72d1b37ce6 WatchSource:0}: Error finding container 3fe9b84c2c494c46a0d9f69fa8d431365d83287e0bef4f201c5f3e72d1b37ce6: Status 404 returned error can't find the container with id 3fe9b84c2c494c46a0d9f69fa8d431365d83287e0bef4f201c5f3e72d1b37ce6
Nov 01 08:49:54.708095 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.708064    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:54.708180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.708128    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:54.708180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.708154    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:54.752798 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.752772    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"]
Nov 01 08:49:54.753594 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:49:54.753570    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4636a672_4b3c_4f31_b8b6_3eef2dde47db.slice/crio-dc80d8f7423bb8a18a3c7e912a93137e9f3ba411904e3125c5eb4b6637bffbb9 WatchSource:0}: Error finding container dc80d8f7423bb8a18a3c7e912a93137e9f3ba411904e3125c5eb4b6637bffbb9: Status 404 returned error can't find the container with id dc80d8f7423bb8a18a3c7e912a93137e9f3ba411904e3125c5eb4b6637bffbb9
Nov 01 08:49:54.755939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.755910    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:54.756024 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.755982    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:54.756094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:54.756025    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:49:55.429612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:55.429569    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" event={"ID":"4636a672-4b3c-4f31-b8b6-3eef2dde47db","Type":"ContainerStarted","Data":"342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61"}
Nov 01 08:49:55.430103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:55.429664    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" event={"ID":"4636a672-4b3c-4f31-b8b6-3eef2dde47db","Type":"ContainerStarted","Data":"dc80d8f7423bb8a18a3c7e912a93137e9f3ba411904e3125c5eb4b6637bffbb9"}
Nov 01 08:49:55.431212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:55.431183    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" event={"ID":"bf477d9c-9a2f-4618-998a-150e6baa9147","Type":"ContainerStarted","Data":"9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0"}
Nov 01 08:49:55.431311 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:55.431231    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" event={"ID":"bf477d9c-9a2f-4618-998a-150e6baa9147","Type":"ContainerStarted","Data":"3fe9b84c2c494c46a0d9f69fa8d431365d83287e0bef4f201c5f3e72d1b37ce6"}
Nov 01 08:49:55.544914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:55.544864    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podStartSLOduration=2.544849772 podStartE2EDuration="2.544849772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:55.543631117 +0000 UTC m=+1068.738025670" watchObservedRunningTime="2026-11-01 08:49:55.544849772 +0000 UTC m=+1068.739244325"
Nov 01 08:49:55.678559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:55.658048    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podStartSLOduration=2.658030473 podStartE2EDuration="2.658030473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:55.656856346 +0000 UTC m=+1068.851250899" watchObservedRunningTime="2026-11-01 08:49:55.658030473 +0000 UTC m=+1068.852425027"
Nov 01 08:49:56.291642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:56.291600    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:49:56.292765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:56.292734    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:49:56.292971 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:56.292944    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:49:56.335013 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:56.334980    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:49:56.336324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:56.336300    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:49:56.336478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:56.336461    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:49:57.290555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:57.289900    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:49:57.290555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:57.289963    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:49:57.335152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:57.335120    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:49:57.335319 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:57.335172    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:49:57.414012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:57.413982    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:49:58.303567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:58.302931    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:49:58.303567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:58.303003    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:49:58.334391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:58.334359    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:49:58.334566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:58.334411    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:49:59.291992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:59.291954    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:49:59.292296 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:59.292267    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:49:59.334336 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:59.334302    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:49:59.334738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:49:59.334362    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:50:00.290558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:00.289930    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:50:00.290558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:00.289993    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:50:00.334339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:00.334298    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:50:00.334761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:00.334356    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:50:01.290059 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:01.290018    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:50:01.290278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:01.290071    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:50:01.334847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:01.334808    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:50:01.335327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:01.334862    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:50:02.290808 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:02.290768    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:50:02.290988 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:02.290843    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 08:50:02.297604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:02.297576    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:02.334769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:02.334733    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:50:02.334985 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:02.334797    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 01 08:50:03.230093 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:03.230041    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:03.376616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:03.376576    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:04.119466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:04.119432    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:50:04.120070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:04.120046    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:50:04.121024 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:04.121005    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 08:50:04.204329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:04.204297    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:50:04.205179 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:04.205154    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:50:04.205864 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:04.205844    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 08:50:05.812729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:05.812692    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:07.088027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:07.087993    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:11.323061 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:11.323017    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:50:11.889436 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:11.889403    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:16.693306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:16.693248    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:16.892727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:16.892688    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:21.503358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:21.503302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:26.393883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:26.393836    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:29.855435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:29.855399    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"]
Nov 01 08:50:29.880811 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:29.880782    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.161616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.095818    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:30.407823 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.224103    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"distinct-w4-b711ae-kserve-self-signed-certs\""
Nov 01 08:50:30.420608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.289899    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.420919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.420888    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.421099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.421079    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvxr\" (UniqueName: \"kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.421418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.421328    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.421670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.421648    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.421861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.421844    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.437553 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.407594    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"]
Nov 01 08:50:30.528721 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.528679    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.536717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.536687    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.537318 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.537296    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.537501 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.537483    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.541717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.541694    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.542255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.542234    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvxr\" (UniqueName: \"kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.542822 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.537197    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.542955 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.538976    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.543050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.529579    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.581297 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.581265    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.648562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.647940    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.648562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.648378    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvxr\" (UniqueName: \"kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr\") pod \"distinct-w4-b711ae-kserve-5db4856578-ftx66\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:30.708945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:30.708848    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:31.719169 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:31.719136    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"]
Nov 01 08:50:31.752636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:31.752591    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:31.902284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:31.902240    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" event={"ID":"47c20553-56d3-442c-971c-4864a829112e","Type":"ContainerStarted","Data":"ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e"}
Nov 01 08:50:31.903619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:31.903592    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" event={"ID":"47c20553-56d3-442c-971c-4864a829112e","Type":"ContainerStarted","Data":"dd061363a2b0c0b6456dbdb12365d41949b37907d23098cb2396541e9bfe2738"}
Nov 01 08:50:31.933932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:31.933893    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:50:31.934117 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:31.933960    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:50:32.012313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:32.012254    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podStartSLOduration=3.012237139 podStartE2EDuration="3.012237139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:32.011084147 +0000 UTC m=+1105.205478706" watchObservedRunningTime="2026-11-01 08:50:32.012237139 +0000 UTC m=+1105.206631673"
Nov 01 08:50:32.710196 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:32.710163    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:35.000965 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.000928    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2t8v4-2m2gx"]
Nov 01 08:50:35.004905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.004887    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.055240 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.055214    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q5m6c\""
Nov 01 08:50:35.068801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.068769    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2t8v4-2m2gx"]
Nov 01 08:50:35.089771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.089734    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.089941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.089792    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.089941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.089916    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.090049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.090030    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbxd\" (UniqueName: \"kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.100525 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.100499    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 08:50:35.190716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.190679    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.190716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.190728    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.191049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.190780    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.191049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.190843    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbxd\" (UniqueName: \"kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.191257 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.191232    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.191478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.191460    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.240055 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.240019    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 08:50:35.241893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.241869    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.248110 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.248083    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbxd\" (UniqueName: \"kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd\") pod \"periodic-gathering-2t8v4-2m2gx\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") " pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.317020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.316933    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:50:35.684305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.684273    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2t8v4-2m2gx"]
Nov 01 08:50:35.685032 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:50:35.685008    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8630ea6e_3d5a_4391_9d7d_63d9fefec7ec.slice/crio-e289b24a2c84b85c8e0d18d7af3bf8cfd2e77da7b96508110a77d5720bee20d9 WatchSource:0}: Error finding container e289b24a2c84b85c8e0d18d7af3bf8cfd2e77da7b96508110a77d5720bee20d9: Status 404 returned error can't find the container with id e289b24a2c84b85c8e0d18d7af3bf8cfd2e77da7b96508110a77d5720bee20d9
Nov 01 08:50:35.929699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.929665    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx" event={"ID":"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec","Type":"ContainerStarted","Data":"71c544aa1dc9834074f9d046394bdc6e12db80a96135708b17ce87455decf5d5"}
Nov 01 08:50:35.929871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:35.929714    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx" event={"ID":"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec","Type":"ContainerStarted","Data":"e289b24a2c84b85c8e0d18d7af3bf8cfd2e77da7b96508110a77d5720bee20d9"}
Nov 01 08:50:36.037837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:36.037775    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx" podStartSLOduration=2.037760085 podStartE2EDuration="2.037760085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:36.037583106 +0000 UTC m=+1109.231977658" watchObservedRunningTime="2026-11-01 08:50:36.037760085 +0000 UTC m=+1109.232154676"
Nov 01 08:50:36.613210 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:36.613172    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:40.709881 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:40.709846    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:40.722429 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:40.722402    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:40.971154 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:40.971062    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:50:41.323322 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:41.323284    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:50:41.378454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:41.378408    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:45.011342 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:45.011302    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:45.011342 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:45.011333    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:45.011885 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:45.011424    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:50:45.51140226 +0000 UTC m=+1118.705796806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:50:45.521173 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:45.521138    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:45.521173 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:45.521169    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:45.521385 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:45.521220    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:50:46.521206155 +0000 UTC m=+1119.715600689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:50:46.179395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:46.179360    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:46.534164 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:46.534126    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:46.534164 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:46.534156    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:46.534396 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:46.534209    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:50:48.534194745 +0000 UTC m=+1121.728589279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:50:47.999902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:47.999864    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"]
Nov 01 08:50:48.005366 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.005340    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.091758 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.091714    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:48.100150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.100113    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"distinct-w2-343207-kserve-self-signed-certs\""
Nov 01 08:50:48.103643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.103626    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"]
Nov 01 08:50:48.157323 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.157282    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.157323 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.157333    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6pm\" (UniqueName: \"kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.157632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.157492    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.157632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.157605    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.157729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.157702    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.157885 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.157861    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.258954    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259005    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259234 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259066    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259234 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259181    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259234 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259205    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6pm\" (UniqueName: \"kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259238    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259462 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259434    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259470    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.259622 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.259519    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.261748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.261726    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.261960 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.261942    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.316452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.316420    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6pm\" (UniqueName: \"kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm\") pod \"distinct-w2-343207-kserve-7658d67596-cb6nf\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") " pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.317239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.317209    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:48.575698 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:48.575611    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:48.575698 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:48.575641    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:48.575886 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:48.575704    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:50:52.575688237 +0000 UTC m=+1125.770082774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:50:48.720847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:48.720820    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"]
Nov 01 08:50:48.721931 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:50:48.721905    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod858111e0_6d9c_4a65_8a50_ce5d1a448001.slice/crio-053ffe3cce81c04b0dd5e960f6168f4b85df8dd0a249cda46075b3f5fd727024 WatchSource:0}: Error finding container 053ffe3cce81c04b0dd5e960f6168f4b85df8dd0a249cda46075b3f5fd727024: Status 404 returned error can't find the container with id 053ffe3cce81c04b0dd5e960f6168f4b85df8dd0a249cda46075b3f5fd727024
Nov 01 08:50:49.013364 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:49.013326    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" event={"ID":"858111e0-6d9c-4a65-8a50-ce5d1a448001","Type":"ContainerStarted","Data":"f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93"}
Nov 01 08:50:49.013821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:49.013375    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" event={"ID":"858111e0-6d9c-4a65-8a50-ce5d1a448001","Type":"ContainerStarted","Data":"053ffe3cce81c04b0dd5e960f6168f4b85df8dd0a249cda46075b3f5fd727024"}
Nov 01 08:50:49.125133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:49.125078    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podStartSLOduration=2.125064992 podStartE2EDuration="2.125064992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:49.122285987 +0000 UTC m=+1122.316680542" watchObservedRunningTime="2026-11-01 08:50:49.125064992 +0000 UTC m=+1122.319459546"
Nov 01 08:50:49.317672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:49.317584    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:51.107985 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:51.107951    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:52.620905 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:52.620871    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:52.620905 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:52.620901    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:50:52.621355 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:50:52.620961    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:51:00.620945047 +0000 UTC m=+1133.815339580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:50:55.989930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:55.989885    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:57.880054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:57.880013    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:58.030774 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:58.030744    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:58.187385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:58.187297    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:58.318166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:58.318127    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:58.330786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:58.330760    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:58.985360 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:58.985329    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"]
Nov 01 08:50:58.988833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:58.988815    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.128889    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.128987    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129072    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129110    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129155    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129233    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129304    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrzt6\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129385    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.129571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.129436    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.132813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.132670    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"]
Nov 01 08:50:59.135577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.135554    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:50:59.230877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.230833    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.230923    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.230981    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231046    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231287 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231247    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231313    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231328    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231438    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231505    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzt6\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231651    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.231829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.231819    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.232180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.232158    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.232285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.232258    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.232432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.232407    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.234170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.234146    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.234595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.234573    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.288867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.288794    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzt6\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.288867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.288856    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.302879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.302850    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:50:59.722917 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.722887    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"]
Nov 01 08:50:59.724083 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:50:59.724052    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012254b6_ea0f_4d47_a50e_80da612a43f1.slice/crio-59d553452f301a9642e2e0fd99607249ffecd877c291fa2ce5f7dd2e1b3c5183 WatchSource:0}: Error finding container 59d553452f301a9642e2e0fd99607249ffecd877c291fa2ce5f7dd2e1b3c5183: Status 404 returned error can't find the container with id 59d553452f301a9642e2e0fd99607249ffecd877c291fa2ce5f7dd2e1b3c5183
Nov 01 08:50:59.726282 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.726247    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:50:59.726358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.726314    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:50:59.726358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.726343    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:50:59.776866 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.776838    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:50:59.925332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:50:59.925301    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:00.111232 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.111192    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" event={"ID":"012254b6-ea0f-4d47-a50e-80da612a43f1","Type":"ContainerStarted","Data":"0fcd56093e21a675a1c16afefb97c14f084db6044d1127a0cfd254d0d93169a8"}
Nov 01 08:51:00.111778 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.111253    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" event={"ID":"012254b6-ea0f-4d47-a50e-80da612a43f1","Type":"ContainerStarted","Data":"59d553452f301a9642e2e0fd99607249ffecd877c291fa2ce5f7dd2e1b3c5183"}
Nov 01 08:51:00.227391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.227321    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podStartSLOduration=2.22730734 podStartE2EDuration="2.22730734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:00.223864048 +0000 UTC m=+1133.418258603" watchObservedRunningTime="2026-11-01 08:51:00.22730734 +0000 UTC m=+1133.421701959"
Nov 01 08:51:00.303951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.303920    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:51:00.305334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.305302    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:00.305460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.305359    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:00.474635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.471042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:00.622935 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.622904    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:00.650114 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:00.650086    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:51:00.650114 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:00.650110    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:51:00.650314 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:00.650171    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:51:16.650153296 +0000 UTC m=+1149.844547829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:51:00.893949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:00.893914    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:01.303851 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:01.303820    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:01.304246 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:01.303868    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:02.304260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:02.304214    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:02.304774 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:02.304272    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:02.674301 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:02.674126    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:03.303869 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:03.303836    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:03.304043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:03.303891    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:04.303445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:04.303411    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:04.303846 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:04.303463    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:04.436948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:04.436895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:04.693764 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:04.693651    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:05.304031 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:05.303988    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:05.304650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:05.304615    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:05.934193 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:05.934157    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:06.197842 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:06.197738    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:06.303689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:06.303655    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:51:06.303880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:06.303713    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 08:51:07.127005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.126966    4314 scope.go:122] "RemoveContainer" containerID="10a9b7d45fdb451719e5f245c9a7d908685ef63a9ab8b9249c09ac4418f8fbdd"
Nov 01 08:51:07.139238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.139212    4314 scope.go:122] "RemoveContainer" containerID="511053eadd95800309d3ff1118d70e9bcbbc4b7cae56450f06cb35e4d1aed4c0"
Nov 01 08:51:07.150587 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.150517    4314 scope.go:122] "RemoveContainer" containerID="4bf61e74d5bf6f3020f473fa658a01bed225b1ce85f1cb7b7e24d0f0ec91f99e"
Nov 01 08:51:07.160574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.160481    4314 scope.go:122] "RemoveContainer" containerID="8d361d268ec8b2f864997f3a2b6276a6eecc4cce22d96fb0f49ad972a08f8833"
Nov 01 08:51:07.755687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.755651    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:51:07.756061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.755994    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:51:07.758738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:07.758690    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 08:51:10.803855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:10.803820    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:11.321665 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:11.321626    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:51:14.417211 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:14.417174    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"]
Nov 01 08:51:15.729343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:15.729168    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:16.711835 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:16.711805    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f77f55536278: secret "agc-cred-f77f55536278" not found
Nov 01 08:51:16.712084 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:16.712067    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6: secret "agc-cred-f77f55536278" not found
Nov 01 08:51:16.712329 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:16.712314    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials podName:0e52afd4-7270-4dd9-bae3-8df8e5938bd0 nodeName:}" failed. No retries permitted until 2026-11-01 08:51:48.712259036 +0000 UTC m=+1181.906653584 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials") pod "payload-processing-external-model-555b8b678f-ptnh6" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0") : secret "agc-cred-f77f55536278" not found
Nov 01 08:51:16.950556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:16.935630    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:51:16.950556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:16.935732    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:19.423156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.423120    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6" podUID="0e52afd4-7270-4dd9-bae3-8df8e5938bd0" containerName="payload-processing" containerID="cri-o://12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35" gracePeriod=25
Nov 01 08:51:19.681624 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.681560    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:51:19.847030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.846997    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm\") pod \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") "
Nov 01 08:51:19.847212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847061    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume\") pod \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") "
Nov 01 08:51:19.847212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847158    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume\") pod \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") "
Nov 01 08:51:19.847212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847199    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials\") pod \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\" (UID: \"0e52afd4-7270-4dd9-bae3-8df8e5938bd0\") "
Nov 01 08:51:19.847485 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847461    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume" pod "0e52afd4-7270-4dd9-bae3-8df8e5938bd0" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:51:19.847567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847494    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume" pod "0e52afd4-7270-4dd9-bae3-8df8e5938bd0" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:51:19.847668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847649    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:51:19.847719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.847674    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:51:19.849284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.849264    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm" pod "0e52afd4-7270-4dd9-bae3-8df8e5938bd0" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0"). InnerVolumeSpecName "kube-api-access-vbvjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:51:19.849383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.849364    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials" pod "0e52afd4-7270-4dd9-bae3-8df8e5938bd0" (UID: "0e52afd4-7270-4dd9-bae3-8df8e5938bd0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:51:19.949073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.948994    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:51:19.949073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:19.949023    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvjm\" (UniqueName: \"kubernetes.io/projected/0e52afd4-7270-4dd9-bae3-8df8e5938bd0-kube-api-access-vbvjm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:51:20.308396 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.308356    4314 generic.go:356] "Generic (PLEG): container finished" podID="0e52afd4-7270-4dd9-bae3-8df8e5938bd0" containerID="12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35" exitCode=0
Nov 01 08:51:20.308615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.308405    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"
Nov 01 08:51:20.308615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.308442    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6" event={"ID":"0e52afd4-7270-4dd9-bae3-8df8e5938bd0","Type":"ContainerDied","Data":"12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35"}
Nov 01 08:51:20.308615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.308497    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6" event={"ID":"0e52afd4-7270-4dd9-bae3-8df8e5938bd0","Type":"ContainerDied","Data":"de307ceeb421fcab596b13b9ecbb302fabcf44661be9cabc57ac645c155bce0b"}
Nov 01 08:51:20.308615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.308524    4314 scope.go:122] "RemoveContainer" containerID="12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35"
Nov 01 08:51:20.318823 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.318802    4314 scope.go:122] "RemoveContainer" containerID="12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35"
Nov 01 08:51:20.319126 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:20.319105    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35\": container with ID starting with 12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35 not found: ID does not exist" containerID="12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35"
Nov 01 08:51:20.319177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.319136    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35"} err="failed to get container status \"12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35\": rpc error: code = NotFound desc = could not find container \"12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35\": container with ID starting with 12f3cd3e8fa3cc032401f068fada199f70c1bf37b2084337b4c6a9c95d6fdc35 not found: ID does not exist"
Nov 01 08:51:20.462818 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.462786    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"]
Nov 01 08:51:20.508778 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.508739    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-555b8b678f-ptnh6"]
Nov 01 08:51:20.879313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.879274    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:20.928923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:20.928886    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e52afd4-7270-4dd9-bae3-8df8e5938bd0" path="/var/lib/kubelet/pods/0e52afd4-7270-4dd9-bae3-8df8e5938bd0/volumes"
Nov 01 08:51:25.694838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:25.694805    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:26.681994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.681944    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"]
Nov 01 08:51:26.683099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.682974    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0e52afd4-7270-4dd9-bae3-8df8e5938bd0" containerName="payload-processing"
Nov 01 08:51:26.683238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.683101    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e52afd4-7270-4dd9-bae3-8df8e5938bd0" containerName="payload-processing"
Nov 01 08:51:26.683313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.683267    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e52afd4-7270-4dd9-bae3-8df8e5938bd0" containerName="payload-processing"
Nov 01 08:51:26.687635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.687612    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.725779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.725741    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.725807    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.725919    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.726044    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.726092    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.726242    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.726305    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.726357    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.726576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.726490    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.795689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.795648    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"]
Nov 01 08:51:26.866177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866124    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866233    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866266    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866384    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866456    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866489    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866586    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866628    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.866737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.866678    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.868056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.867845    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.868056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.867976    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.868238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.868163    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.868452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.868426    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.868775 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.868751    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.871150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.871104    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.873594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.873567    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.929224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.929190    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:26.933043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:26.932957    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:27.007930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:27.007879    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:27.470113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:27.469932    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"]
Nov 01 08:51:27.470989 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:51:27.470951    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433f22be_eed2_465f_a782_ecdb78dee65d.slice/crio-f9d3a6650b85625a2f895beec66bfc784ac2285e9c0ecc8116d52c3a9cf3c837 WatchSource:0}: Error finding container f9d3a6650b85625a2f895beec66bfc784ac2285e9c0ecc8116d52c3a9cf3c837: Status 404 returned error can't find the container with id f9d3a6650b85625a2f895beec66bfc784ac2285e9c0ecc8116d52c3a9cf3c837
Nov 01 08:51:27.473911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:27.473863    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:51:27.474018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:27.473961    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:51:27.474018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:27.474008    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:51:28.394557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:28.393975    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" event={"ID":"433f22be-eed2-465f-a782-ecdb78dee65d","Type":"ContainerStarted","Data":"6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674"}
Nov 01 08:51:28.394557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:28.394050    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" event={"ID":"433f22be-eed2-465f-a782-ecdb78dee65d","Type":"ContainerStarted","Data":"f9d3a6650b85625a2f895beec66bfc784ac2285e9c0ecc8116d52c3a9cf3c837"}
Nov 01 08:51:28.645029 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:28.641325    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podStartSLOduration=2.641301938 podStartE2EDuration="2.641301938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:28.622989125 +0000 UTC m=+1161.817383691" watchObservedRunningTime="2026-11-01 08:51:28.641301938 +0000 UTC m=+1161.835696495"
Nov 01 08:51:29.012558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.011693    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:29.014561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.013906    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:29.014561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.013980    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:29.325615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.324707    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:29.401557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.400618    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:29.450565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.448651    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:29.579614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.572651    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:29.757559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.756632    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Nov 01 08:51:29.965615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.964640    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:29.965803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:29.965732    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:51:30.014557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.014039    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:30.014557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.014102    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:30.110729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.110672    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:30.405700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.405651    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:51:30.407520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.407498    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:51:30.451717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.451669    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:51:30.452114 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.452088    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:51:30.598556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.598486    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:51:30.634655 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.607156    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:51:30.738623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.711837    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:30.961782 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.865440    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:30.985556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.965292    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:30.985556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:30.974966    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:31.023064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.022973    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:31.023343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.023312    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:31.100577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.093517    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:51:31.100577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.093596    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:51:31.119558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.113199    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:51:31.512563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.512383    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:51:31.512563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.512491    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:51:31.525650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.520347    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:51:31.525650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.522216    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:51:31.629416 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.629364    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:31.640714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.638354    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:51:31.657595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.657564    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:51:31.825721 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.825627    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:51:31.839605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:31.839514    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:51:32.009884 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:32.009849    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:32.010050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:32.009902    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:33.009153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:33.009119    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:33.009604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:33.009172    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:34.008916 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:34.008881    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:34.009096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:34.008937    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:35.008516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:35.008479    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:51:35.008956 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:35.008567    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 08:51:35.689689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:35.689657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:36.427201 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:36.427134    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:36.428073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:36.428051    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:36.430075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:36.430048    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 08:51:40.498938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:40.498897    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:41.323239 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:51:41.323201    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:51:41.403260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.403226    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:41.812045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.812011    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"]
Nov 01 08:51:41.817323 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.817299    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973295 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973260    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973451 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973409    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973459    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973600 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973513    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973747 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973712    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973887 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973839    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.973954 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973924    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.974014 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.973986    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrblz\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:41.974110 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:41.974082    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.014795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.014761    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"]
Nov 01 08:51:42.075572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075459    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.075742 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075575    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.075742 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075660    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrblz\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.075742 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075698    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075739    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075803    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075830    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075853    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.075905    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.076074    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.076130    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.076229    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076284 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.076265    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.076636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.076603    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.078042 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.078020    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.078391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.078370    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.132976 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.132937    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.133112 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.133089    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrblz\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.432165 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.432091    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:42.851419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.851386    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"]
Nov 01 08:51:42.852843 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:51:42.852812    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6963b4_79b3_488f_a6c3_3fceacb5b459.slice/crio-6c3e5f327fd9b9de5781cd16452645a2813568d72c0d48ccc89b443a56350553 WatchSource:0}: Error finding container 6c3e5f327fd9b9de5781cd16452645a2813568d72c0d48ccc89b443a56350553: Status 404 returned error can't find the container with id 6c3e5f327fd9b9de5781cd16452645a2813568d72c0d48ccc89b443a56350553
Nov 01 08:51:42.855371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.855339    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:51:42.855462 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.855403    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:51:42.855462 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:42.855434    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:51:43.689786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:43.689742    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" event={"ID":"6e6963b4-79b3-488f-a6c3-3fceacb5b459","Type":"ContainerStarted","Data":"2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf"}
Nov 01 08:51:43.689980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:43.689848    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" event={"ID":"6e6963b4-79b3-488f-a6c3-3fceacb5b459","Type":"ContainerStarted","Data":"6c3e5f327fd9b9de5781cd16452645a2813568d72c0d48ccc89b443a56350553"}
Nov 01 08:51:43.852995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:43.852941    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podStartSLOduration=2.852928246 podStartE2EDuration="2.852928246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:43.850580113 +0000 UTC m=+1177.044974668" watchObservedRunningTime="2026-11-01 08:51:43.852928246 +0000 UTC m=+1177.047322801"
Nov 01 08:51:44.432978 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:44.432944    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:44.434512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:44.434485    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:51:44.434652 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:44.434559    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 08:51:45.432765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:45.432729    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:51:45.433200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:45.432787    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 08:51:45.609887 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:45.609852    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:46.433352 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:46.433311    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:51:46.433793 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:46.433438    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 08:51:47.432964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:47.432926    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:51:47.433152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:47.432990    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 08:51:48.433205 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:48.433166    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:51:48.433706 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:48.433231    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 08:51:49.433274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:49.433234    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:51:49.433847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:49.433302    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 08:51:50.514521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:50.514488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:51:50.517183 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:50.517156    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:50.517773 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:50.517745    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:50.518920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:50.518901    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 08:51:55.386337 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:51:55.386296    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:00.183501 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:00.183457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:00.545297 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:00.545256    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:04.991346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:04.991313    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:06.525028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:06.524990    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:09.768877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:09.768826    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:11.336969 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:52:11.336932    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:52:11.881625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:11.881476    4314 generic.go:356] "Generic (PLEG): container finished" podID="8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" containerID="71c544aa1dc9834074f9d046394bdc6e12db80a96135708b17ce87455decf5d5" exitCode=0
Nov 01 08:52:11.882187 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:11.882137    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx" event={"ID":"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec","Type":"ContainerDied","Data":"71c544aa1dc9834074f9d046394bdc6e12db80a96135708b17ce87455decf5d5"}
Nov 01 08:52:13.049644 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.049617    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:52:13.169891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.169792    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle\") pod \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") "
Nov 01 08:52:13.169891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.169882    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path\") pod \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") "
Nov 01 08:52:13.170129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.169983    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd\" (UniqueName: \"kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd\") pod \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") "
Nov 01 08:52:13.170129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.170080    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle\") pod \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\" (UID: \"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec\") "
Nov 01 08:52:13.170266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.170227    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle" pod "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" (UID: "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:52:13.170378 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.170288    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path" pod "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" (UID: "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:52:13.170544 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.170500    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle" pod "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" (UID: "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:52:13.171066 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.171047    4314 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-service-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:13.171066 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.171067    4314 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-archives-path\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:13.171222 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.171077    4314 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-trusted-ca-bundle\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:13.172157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.172138    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd" pod "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" (UID: "8630ea6e-3d5a-4391-9d7d-63d9fefec7ec"). InnerVolumeSpecName "kube-api-access-khbxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:52:13.272914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.272875    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbxd\" (UniqueName: \"kubernetes.io/projected/8630ea6e-3d5a-4391-9d7d-63d9fefec7ec-kube-api-access-khbxd\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:13.895914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.895886    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx"
Nov 01 08:52:13.895914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.895893    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2t8v4-2m2gx" event={"ID":"8630ea6e-3d5a-4391-9d7d-63d9fefec7ec","Type":"ContainerDied","Data":"e289b24a2c84b85c8e0d18d7af3bf8cfd2e77da7b96508110a77d5720bee20d9"}
Nov 01 08:52:13.896128 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:13.895964    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e289b24a2c84b85c8e0d18d7af3bf8cfd2e77da7b96508110a77d5720bee20d9"
Nov 01 08:52:14.581058 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:14.581015    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:15.187161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:15.187125    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:15.332390 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:15.332359    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:18.468010 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.467970    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"]
Nov 01 08:52:18.468520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.468505    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" containerName="insights-gathering"
Nov 01 08:52:18.468601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.468520    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" containerName="insights-gathering"
Nov 01 08:52:18.468648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.468638    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8630ea6e-3d5a-4391-9d7d-63d9fefec7ec" containerName="insights-gathering"
Nov 01 08:52:18.472370 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.472348    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.513906 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.513873    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"]
Nov 01 08:52:18.517739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.517722    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.540121 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540091    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540144    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540203    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540264    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540310    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540417 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540360    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540417 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540402    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540479 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540422    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.540516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.540485    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgns\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.567419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.567387    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"]
Nov 01 08:52:18.614670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.614636    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"]
Nov 01 08:52:18.641858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.641795    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.641858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.641833    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.641916    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.641958    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.641988    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642033    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642049    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642076    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642097    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642128    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kkqx\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642154    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642181    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642204    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642251    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642271    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgns\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642329    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642439    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.642999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642455    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.642999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.642825    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.643276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.643179    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.643276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.643250    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.643723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.643684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.643834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.643791    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.645511 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.645484    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.645915 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.645889    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.700729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.700662    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.700863 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.700797    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgns\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.745112 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745079    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kkqx\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745170    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745226    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745293    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745387 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745354    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745385    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745428    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745484    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745509    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745881 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745830    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.745881 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745865    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.746049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745922    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.746049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.745999    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.746285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.746263    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.747598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.747575    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.747800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.747783    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.787231 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.787196    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:18.802124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.802099    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.802613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.802588    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kkqx\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:18.831473 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:18.831436    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:19.205888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.205800    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"]
Nov 01 08:52:19.239936 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:19.239906    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd694432c_21e5_4c92_be3f_d4674d18b15b.slice/crio-355f84a75dc7a49ba7e360ae9a6e748f392bdfd6d1ecae7aee666bb7a0093dfc WatchSource:0}: Error finding container 355f84a75dc7a49ba7e360ae9a6e748f392bdfd6d1ecae7aee666bb7a0093dfc: Status 404 returned error can't find the container with id 355f84a75dc7a49ba7e360ae9a6e748f392bdfd6d1ecae7aee666bb7a0093dfc
Nov 01 08:52:19.242353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.242318    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:19.242459 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.242405    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:19.242520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.242450    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:19.258962 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:19.258927    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35061eb7_9af6_46fd_a0f1_f3358f762de6.slice/crio-5c4885d574c0fe02fb8e307d205e8e2ad0f8b0745c170a853e79483caab690c7 WatchSource:0}: Error finding container 5c4885d574c0fe02fb8e307d205e8e2ad0f8b0745c170a853e79483caab690c7: Status 404 returned error can't find the container with id 5c4885d574c0fe02fb8e307d205e8e2ad0f8b0745c170a853e79483caab690c7
Nov 01 08:52:19.259912 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.259890    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"]
Nov 01 08:52:19.261264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.261233    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:19.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.261294    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:19.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.261329    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:19.435079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.435043    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:19.937347 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.937313    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" event={"ID":"35061eb7-9af6-46fd-a0f1-f3358f762de6","Type":"ContainerStarted","Data":"e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842"}
Nov 01 08:52:19.937815 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.937369    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" event={"ID":"35061eb7-9af6-46fd-a0f1-f3358f762de6","Type":"ContainerStarted","Data":"5c4885d574c0fe02fb8e307d205e8e2ad0f8b0745c170a853e79483caab690c7"}
Nov 01 08:52:19.939343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.939317    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" event={"ID":"d694432c-21e5-4c92-be3f-d4674d18b15b","Type":"ContainerStarted","Data":"23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db"}
Nov 01 08:52:19.939463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:19.939366    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" event={"ID":"d694432c-21e5-4c92-be3f-d4674d18b15b","Type":"ContainerStarted","Data":"355f84a75dc7a49ba7e360ae9a6e748f392bdfd6d1ecae7aee666bb7a0093dfc"}
Nov 01 08:52:20.096875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.096807    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podStartSLOduration=2.096783544 podStartE2EDuration="2.096783544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:20.075590307 +0000 UTC m=+1213.269984862" watchObservedRunningTime="2026-11-01 08:52:20.096783544 +0000 UTC m=+1213.291178099"
Nov 01 08:52:20.211450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.211327    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podStartSLOduration=2.211311513 podStartE2EDuration="2.211311513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:20.204675396 +0000 UTC m=+1213.399069953" watchObservedRunningTime="2026-11-01 08:52:20.211311513 +0000 UTC m=+1213.405706066"
Nov 01 08:52:20.789555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.789370    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:20.789772 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.789588    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:52:20.789772 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.789643    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 08:52:20.831900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.831869    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:20.833002 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.832974    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:52:20.833135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.833041    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 08:52:20.952512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:20.952478    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:21.787908 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:21.787871    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:52:21.788234 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:21.788193    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 08:52:21.832009 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:21.831980    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:52:21.832169 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:21.832026    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 08:52:22.789556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:22.788857    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:52:22.789556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:22.788918    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 08:52:22.831996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:22.831965    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:52:22.832158 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:22.832010    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 08:52:23.793955 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:23.793916    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:52:23.794571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:23.794514    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 08:52:23.833706 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:23.833679    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:52:23.833939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:23.833917    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 08:52:24.408211 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:24.408170    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:24.788352 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:24.787875    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:52:24.788352 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:24.787946    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 08:52:24.832167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:24.832121    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:52:24.832598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:24.832183    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 08:52:25.091613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:25.091350    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:25.788172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:25.788133    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:52:25.788380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:25.788207    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 08:52:25.832855 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:25.832814    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:52:25.833253 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:25.832881    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 08:52:27.070170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:27.070139    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:27.070993 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:27.070965    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:27.071763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:27.071741    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 08:52:27.080082 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:27.080055    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:27.080386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:27.080362    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:27.081107 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:27.081085    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 08:52:29.247621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.247587    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"]
Nov 01 08:52:29.251573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.251550    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.287688 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.287654    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:29.335374 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.333016    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"]
Nov 01 08:52:29.367916 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.367878    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.367925    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.367958    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.368009    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.368060    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.368301    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.368334    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.368360    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24fv\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.368525 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.368427    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470143 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470100    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470154    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470274    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470294    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470312    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24fv\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470350    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470502    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470675 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470577    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470675 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470610    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470675 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470618    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470675 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470634    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470675    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.470858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.470736    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.471323 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.471300    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.472604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.472580    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.473032 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.473014    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.527568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.527470    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.527730 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.527685    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24fv\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.572705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.572674    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:29.994124 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:29.994096    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf2af26_0a56_4830_8a91_cfe35c4c1023.slice/crio-2c194e5bec1fb20b01521213ddeb16692898b66338d2cec9bd16f30944371ec8 WatchSource:0}: Error finding container 2c194e5bec1fb20b01521213ddeb16692898b66338d2cec9bd16f30944371ec8: Status 404 returned error can't find the container with id 2c194e5bec1fb20b01521213ddeb16692898b66338d2cec9bd16f30944371ec8
Nov 01 08:52:29.994124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.994103    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"]
Nov 01 08:52:29.996503 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.996468    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:29.996588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.996552    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:29.996588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:29.996584    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:30.056291 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:30.056219    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" event={"ID":"cbf2af26-0a56-4830-8a91-cfe35c4c1023","Type":"ContainerStarted","Data":"2c194e5bec1fb20b01521213ddeb16692898b66338d2cec9bd16f30944371ec8"}
Nov 01 08:52:31.090489 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:31.090448    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" event={"ID":"cbf2af26-0a56-4830-8a91-cfe35c4c1023","Type":"ContainerStarted","Data":"beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64"}
Nov 01 08:52:31.248979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:31.248917    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podStartSLOduration=2.248901828 podStartE2EDuration="2.248901828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:31.20226109 +0000 UTC m=+1224.396655643" watchObservedRunningTime="2026-11-01 08:52:31.248901828 +0000 UTC m=+1224.443296383"
Nov 01 08:52:31.578559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:31.577451    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:31.583591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:31.582663    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:31.583591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:31.582724    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:32.576046 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:32.576000    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:32.576736 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:32.576701    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:33.574557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.573951    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:33.574557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.574021    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:33.605816 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.605780    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"]
Nov 01 08:52:33.623729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.623695    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.723105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.723064    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.723376 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.723352    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.723619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.723593    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.723790 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.723772    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.723947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.723929    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.724137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.724120    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdh77\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.724296 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.724278    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.724437 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.724402    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.724590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.724573    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.730887 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.730856    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"]
Nov 01 08:52:33.826261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826073    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdh77\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826120    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826146    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826189    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826268    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826294    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826379    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826416    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.826667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.826477    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.827240 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.827212    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.827411 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.827303    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.827522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.827360    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.827696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.827573    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.827844 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.827819    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.829568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.829522    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.830152 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.829789    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.887579 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.887492    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdh77\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.900021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.899949    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:33.943918 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:33.943887    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:34.478555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.477042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:34.487603 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.480973    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"]
Nov 01 08:52:34.487603 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:34.486597    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44ff955_8bb9_481d_94d1_377fdb20961d.slice/crio-601a9dc7a604529e13749839ebc1259820280f17d52c516d93eae046912b0f29 WatchSource:0}: Error finding container 601a9dc7a604529e13749839ebc1259820280f17d52c516d93eae046912b0f29: Status 404 returned error can't find the container with id 601a9dc7a604529e13749839ebc1259820280f17d52c516d93eae046912b0f29
Nov 01 08:52:34.489835 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.489794    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:34.489946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.489874    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:34.489946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.489924    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:34.583861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.582868    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:34.583861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:34.582923    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:35.137524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.137485    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" event={"ID":"e44ff955-8bb9-481d-94d1-377fdb20961d","Type":"ContainerStarted","Data":"bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9"}
Nov 01 08:52:35.137953 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.137578    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" event={"ID":"e44ff955-8bb9-481d-94d1-377fdb20961d","Type":"ContainerStarted","Data":"601a9dc7a604529e13749839ebc1259820280f17d52c516d93eae046912b0f29"}
Nov 01 08:52:35.582031 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.581989    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:35.582316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.582287    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:35.944670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.944579    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:35.946108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.946088    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:35.946222 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:35.946127    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:36.576018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:36.575977    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:36.576671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:36.576638    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:36.944445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:36.944310    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:36.944445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:36.944396    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:37.574559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:37.573956    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 08:52:37.574559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:37.574020    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 08:52:37.944693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:37.944597    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:37.944693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:37.944670    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:38.586039 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:38.586005    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:38.586316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:38.586288    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:38.587185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:38.587164    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 08:52:38.746006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:38.745956    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podStartSLOduration=5.745942091 podStartE2EDuration="5.745942091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:35.271798514 +0000 UTC m=+1228.466193070" watchObservedRunningTime="2026-11-01 08:52:38.745942091 +0000 UTC m=+1231.940336645"
Nov 01 08:52:38.944967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:38.944887    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:38.944967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:38.944951    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:39.628033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:39.627971    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:39.938238 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:39.938157    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"]
Nov 01 08:52:39.942632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:39.942613    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:39.945044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:39.945023    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:39.945345 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:39.945067    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:39.996756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:39.996713    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"]
Nov 01 08:52:40.036521 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.036477    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.036795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.036770    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2qz\" (UniqueName: \"kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.036869 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.036850    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.038051 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.038032    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:52:40.038142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.038033    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:52:40.038142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.038099    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:52:40.038576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.038527    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:52:40.138017 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.137980    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2qz\" (UniqueName: \"kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.138202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.138031    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.138202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.138098    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.138885 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.138858    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.140463 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.140442    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.195765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.195689    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2qz\" (UniqueName: \"kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz\") pod \"maas-api-revoke-sub-ad5da820-gn7tl\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.256633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.256594    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:40.649067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.649037    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"]
Nov 01 08:52:40.649192 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:40.649168    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617e5258_7bc4_4719_8d28_dca8ad3130d7.slice/crio-1c80c4e451df2275cccb6227fe1256823d863ca49d77f9c37a6616df65b58422 WatchSource:0}: Error finding container 1c80c4e451df2275cccb6227fe1256823d863ca49d77f9c37a6616df65b58422: Status 404 returned error can't find the container with id 1c80c4e451df2275cccb6227fe1256823d863ca49d77f9c37a6616df65b58422
Nov 01 08:52:40.945053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.945018    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:40.945466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:40.945074    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:41.202044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.201951    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl" event={"ID":"617e5258-7bc4-4719-8d28-dca8ad3130d7","Type":"ContainerStarted","Data":"2e319a266932d65e54987ad73ff1a8c393924900a8347d9448b63ab6261070d4"}
Nov 01 08:52:41.202044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.202017    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl" event={"ID":"617e5258-7bc4-4719-8d28-dca8ad3130d7","Type":"ContainerStarted","Data":"1c80c4e451df2275cccb6227fe1256823d863ca49d77f9c37a6616df65b58422"}
Nov 01 08:52:41.311689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.311619    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl" podStartSLOduration=2.311597717 podStartE2EDuration="2.311597717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:41.310793334 +0000 UTC m=+1234.505187886" watchObservedRunningTime="2026-11-01 08:52:41.311597717 +0000 UTC m=+1234.505992275"
Nov 01 08:52:41.324838 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:52:41.324734    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:52:41.478418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.478329    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"]
Nov 01 08:52:41.482568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.482545    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.537161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.537122    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"]
Nov 01 08:52:41.557389 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.557341    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6p4\" (UniqueName: \"kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.557603 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.557404    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.557603 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.557482    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.658730 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.658691    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6p4\" (UniqueName: \"kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.658924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.658756    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.658924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.658781    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.659513 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.659484    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.661307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.661273    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.716510 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.716463    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6p4\" (UniqueName: \"kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4\") pod \"maas-api-revoke-sub-1bc086c9-rxzhk\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.797600 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.797565    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:41.945354 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.945319    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 08:52:41.945874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:41.945379    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 08:52:42.182283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.182246    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"]
Nov 01 08:52:42.183023 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:42.182997    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f67cfc_ffe0_422b_b5de_02d9e758bd3f.slice/crio-3fc0b88e45259ee00919fd49d25e76b8cd98fa9b56ee037e9acf7c256ff6bc53 WatchSource:0}: Error finding container 3fc0b88e45259ee00919fd49d25e76b8cd98fa9b56ee037e9acf7c256ff6bc53: Status 404 returned error can't find the container with id 3fc0b88e45259ee00919fd49d25e76b8cd98fa9b56ee037e9acf7c256ff6bc53
Nov 01 08:52:42.213383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.213346    4314 generic.go:356] "Generic (PLEG): container finished" podID="617e5258-7bc4-4719-8d28-dca8ad3130d7" containerID="2e319a266932d65e54987ad73ff1a8c393924900a8347d9448b63ab6261070d4" exitCode=0
Nov 01 08:52:42.213600 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.213433    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl" event={"ID":"617e5258-7bc4-4719-8d28-dca8ad3130d7","Type":"ContainerDied","Data":"2e319a266932d65e54987ad73ff1a8c393924900a8347d9448b63ab6261070d4"}
Nov 01 08:52:42.215245 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.215218    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk" event={"ID":"72f67cfc-ffe0-422b-b5de-02d9e758bd3f","Type":"ContainerStarted","Data":"3fc0b88e45259ee00919fd49d25e76b8cd98fa9b56ee037e9acf7c256ff6bc53"}
Nov 01 08:52:42.949397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.949368    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:42.949869 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.949719    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:42.950633 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:42.950612    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 08:52:43.224048 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.223965    4314 generic.go:356] "Generic (PLEG): container finished" podID="72f67cfc-ffe0-422b-b5de-02d9e758bd3f" containerID="26c3f35b7cfa5ce56762eb3d4f82bc09a0c7a346b7cca7a2ca2bc4c67ccf341b" exitCode=0
Nov 01 08:52:43.224225 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.224043    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk" event={"ID":"72f67cfc-ffe0-422b-b5de-02d9e758bd3f","Type":"ContainerDied","Data":"26c3f35b7cfa5ce56762eb3d4f82bc09a0c7a346b7cca7a2ca2bc4c67ccf341b"}
Nov 01 08:52:43.377625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.377595    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:43.482888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.482858    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca\") pod \"617e5258-7bc4-4719-8d28-dca8ad3130d7\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") "
Nov 01 08:52:43.483045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.482993    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz\" (UniqueName: \"kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz\") pod \"617e5258-7bc4-4719-8d28-dca8ad3130d7\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") "
Nov 01 08:52:43.483045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.483028    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca\") pod \"617e5258-7bc4-4719-8d28-dca8ad3130d7\" (UID: \"617e5258-7bc4-4719-8d28-dca8ad3130d7\") "
Nov 01 08:52:43.483446 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.483421    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca" pod "617e5258-7bc4-4719-8d28-dca8ad3130d7" (UID: "617e5258-7bc4-4719-8d28-dca8ad3130d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:52:43.483607 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.483585    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:43.484981 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.484942    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca" pod "617e5258-7bc4-4719-8d28-dca8ad3130d7" (UID: "617e5258-7bc4-4719-8d28-dca8ad3130d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:52:43.485755 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.485689    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz" pod "617e5258-7bc4-4719-8d28-dca8ad3130d7" (UID: "617e5258-7bc4-4719-8d28-dca8ad3130d7"). InnerVolumeSpecName "kube-api-access-rd2qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:52:43.585306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.585273    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617e5258-7bc4-4719-8d28-dca8ad3130d7-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:43.585306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:43.585304    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2qz\" (UniqueName: \"kubernetes.io/projected/617e5258-7bc4-4719-8d28-dca8ad3130d7-kube-api-access-rd2qz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:44.231753 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.231723    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"
Nov 01 08:52:44.232147 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.231754    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl" event={"ID":"617e5258-7bc4-4719-8d28-dca8ad3130d7","Type":"ContainerDied","Data":"1c80c4e451df2275cccb6227fe1256823d863ca49d77f9c37a6616df65b58422"}
Nov 01 08:52:44.232147 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.231806    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c80c4e451df2275cccb6227fe1256823d863ca49d77f9c37a6616df65b58422"
Nov 01 08:52:44.386461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.386435    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:44.498206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.497478    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca\") pod \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") "
Nov 01 08:52:44.498206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.497618    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4\" (UniqueName: \"kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4\") pod \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") "
Nov 01 08:52:44.498206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.497714    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca\") pod \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\" (UID: \"72f67cfc-ffe0-422b-b5de-02d9e758bd3f\") "
Nov 01 08:52:44.498206 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.498116    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca" pod "72f67cfc-ffe0-422b-b5de-02d9e758bd3f" (UID: "72f67cfc-ffe0-422b-b5de-02d9e758bd3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:52:44.498461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.498386    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:44.499772 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.499745    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca" pod "72f67cfc-ffe0-422b-b5de-02d9e758bd3f" (UID: "72f67cfc-ffe0-422b-b5de-02d9e758bd3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:52:44.500131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.500105    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4" pod "72f67cfc-ffe0-422b-b5de-02d9e758bd3f" (UID: "72f67cfc-ffe0-422b-b5de-02d9e758bd3f"). InnerVolumeSpecName "kube-api-access-jd6p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:52:44.500436 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.500409    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:44.599883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.599851    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:44.599883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:44.599882    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd6p4\" (UniqueName: \"kubernetes.io/projected/72f67cfc-ffe0-422b-b5de-02d9e758bd3f-kube-api-access-jd6p4\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:45.032285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:45.032249    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:45.240103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:45.240068    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk" event={"ID":"72f67cfc-ffe0-422b-b5de-02d9e758bd3f","Type":"ContainerDied","Data":"3fc0b88e45259ee00919fd49d25e76b8cd98fa9b56ee037e9acf7c256ff6bc53"}
Nov 01 08:52:45.240524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:45.240150    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc0b88e45259ee00919fd49d25e76b8cd98fa9b56ee037e9acf7c256ff6bc53"
Nov 01 08:52:45.240524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:45.240078    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"
Nov 01 08:52:49.287053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:49.287020    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:50.686791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.686748    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"]
Nov 01 08:52:50.687651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.687633    4314 cpu_manager.go:414] "containerMap: removing container" podUID="617e5258-7bc4-4719-8d28-dca8ad3130d7" containerName="revoke-keys"
Nov 01 08:52:50.687722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.687654    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617e5258-7bc4-4719-8d28-dca8ad3130d7" containerName="revoke-keys"
Nov 01 08:52:50.687722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.687668    4314 cpu_manager.go:414] "containerMap: removing container" podUID="72f67cfc-ffe0-422b-b5de-02d9e758bd3f" containerName="revoke-keys"
Nov 01 08:52:50.687722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.687676    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f67cfc-ffe0-422b-b5de-02d9e758bd3f" containerName="revoke-keys"
Nov 01 08:52:50.687826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.687801    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="617e5258-7bc4-4719-8d28-dca8ad3130d7" containerName="revoke-keys"
Nov 01 08:52:50.687826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.687813    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f67cfc-ffe0-422b-b5de-02d9e758bd3f" containerName="revoke-keys"
Nov 01 08:52:50.693090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.693062    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.742541 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.742509    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"]
Nov 01 08:52:50.784175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.784143    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:52:50.784428 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.784161    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:52:50.784428 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.784198    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:52:50.787721 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.787702    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:52:50.802391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.802358    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.802575 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.802522    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.802674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.802651    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsl9\" (UniqueName: \"kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.904077 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.904042    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsl9\" (UniqueName: \"kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.904258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.904172    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.904307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.904259    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.904911 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.904889    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.906699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.906677    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:50.961460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:50.961387    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsl9\" (UniqueName: \"kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9\") pod \"maas-api-revoke-sub-b38cb09b-rgkdw\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:51.007563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:51.007493    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:51.381418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:51.381391    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"]
Nov 01 08:52:51.381812 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:51.381786    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f83bcf_9efd_4f1f_b59f_9f818a031154.slice/crio-db7e04d87a13722dedcdbecf8302527c298f6931d2e95b64933255669477bc30 WatchSource:0}: Error finding container db7e04d87a13722dedcdbecf8302527c298f6931d2e95b64933255669477bc30: Status 404 returned error can't find the container with id db7e04d87a13722dedcdbecf8302527c298f6931d2e95b64933255669477bc30
Nov 01 08:52:52.285348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:52.285314    4314 generic.go:356] "Generic (PLEG): container finished" podID="a2f83bcf-9efd-4f1f-b59f-9f818a031154" containerID="bf9f654b3f3c87e441633d8e524ff1d62d4cfda76082c79af19f0daba65859b1" exitCode=0
Nov 01 08:52:52.285770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:52.285397    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw" event={"ID":"a2f83bcf-9efd-4f1f-b59f-9f818a031154","Type":"ContainerDied","Data":"bf9f654b3f3c87e441633d8e524ff1d62d4cfda76082c79af19f0daba65859b1"}
Nov 01 08:52:52.285770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:52.285450    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw" event={"ID":"a2f83bcf-9efd-4f1f-b59f-9f818a031154","Type":"ContainerStarted","Data":"db7e04d87a13722dedcdbecf8302527c298f6931d2e95b64933255669477bc30"}
Nov 01 08:52:53.040383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.040339    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"]
Nov 01 08:52:53.046044 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.046014    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.157148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.157105    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159071    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159336 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159317    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159602    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159707    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159823 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159761    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159823 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159810    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159837    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.159932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.159874    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdq2z\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.246340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.245759    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"]
Nov 01 08:52:53.260984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.260939    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.261161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.261046    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.261161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.261136    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.261290 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.261167    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.261584 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.261554    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.262051 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.262024    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.262134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.262073    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.262631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.262587    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdq2z\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.263242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.263218    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.263591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.263565    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.263669 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.263637    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.263728 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.263673    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.263848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.263773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.264287 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.264256    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.265704 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.265678    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.269456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.269427    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.323072 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.323036    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdq2z\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.327478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.327443    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.378156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.368414    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:53.690557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.654102    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:53.739557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.738757    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca\") pod \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") "
Nov 01 08:52:53.739557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.738841    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\") pod \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") "
Nov 01 08:52:53.739557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.739162    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9\" (UniqueName: \"kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9\") pod \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") "
Nov 01 08:52:53.757574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.742935    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca" pod "a2f83bcf-9efd-4f1f-b59f-9f818a031154" (UID: "a2f83bcf-9efd-4f1f-b59f-9f818a031154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:52:53.817605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.811762    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9" pod "a2f83bcf-9efd-4f1f-b59f-9f818a031154" (UID: "a2f83bcf-9efd-4f1f-b59f-9f818a031154"). InnerVolumeSpecName "kube-api-access-5gsl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:52:53.847104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.846711    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca" pod "a2f83bcf-9efd-4f1f-b59f-9f818a031154" (UID: "a2f83bcf-9efd-4f1f-b59f-9f818a031154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:52:53.847104 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.847066    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\") pod \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\" (UID: \"a2f83bcf-9efd-4f1f-b59f-9f818a031154\") "
Nov 01 08:52:53.847510 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:53.847492    4314 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a2f83bcf-9efd-4f1f-b59f-9f818a031154/volumes/kubernetes.io~secret/maas-api-xks-ca
Nov 01 08:52:53.847713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.847649    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:53.847713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.847676    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsl9\" (UniqueName: \"kubernetes.io/projected/a2f83bcf-9efd-4f1f-b59f-9f818a031154-kube-api-access-5gsl9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:53.847713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:53.847652    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca" pod "a2f83bcf-9efd-4f1f-b59f-9f818a031154" (UID: "a2f83bcf-9efd-4f1f-b59f-9f818a031154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:52:54.077615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.016811    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f83bcf-9efd-4f1f-b59f-9f818a031154-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:52:54.547574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.524821    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw" event={"ID":"a2f83bcf-9efd-4f1f-b59f-9f818a031154","Type":"ContainerDied","Data":"db7e04d87a13722dedcdbecf8302527c298f6931d2e95b64933255669477bc30"}
Nov 01 08:52:54.547574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.524890    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7e04d87a13722dedcdbecf8302527c298f6931d2e95b64933255669477bc30"
Nov 01 08:52:54.547574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.524968    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"
Nov 01 08:52:54.571776 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.571738    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"]
Nov 01 08:52:54.709522 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:52:54.708466    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35b7a28_8f7d_477f_8584_e999ac50b0dd.slice/crio-103fcf043ae4e4ab0ea43e623c444b71c7a21f938b8ccd5ef604492a1cdb4f82 WatchSource:0}: Error finding container 103fcf043ae4e4ab0ea43e623c444b71c7a21f938b8ccd5ef604492a1cdb4f82: Status 404 returned error can't find the container with id 103fcf043ae4e4ab0ea43e623c444b71c7a21f938b8ccd5ef604492a1cdb4f82
Nov 01 08:52:54.810612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.803517    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:54.810612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.803638    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:54.810612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.803691    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:52:54.840940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.823617    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:52:54.851448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:54.845703    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.104015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.103867    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"]
Nov 01 08:52:55.161593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.157662    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.190872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.188889    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d6125e-4bdl6"]
Nov 01 08:52:55.556557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.554351    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.556557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.556282    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:52:55.649559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.644239    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" event={"ID":"a35b7a28-8f7d-477f-8584-e999ac50b0dd","Type":"ContainerStarted","Data":"103fcf043ae4e4ab0ea43e623c444b71c7a21f938b8ccd5ef604492a1cdb4f82"}
Nov 01 08:52:55.671453 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.670656    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.690599 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.689741    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.825555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.824031    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:52:55.825555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.824470    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:52:55.846670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.846619    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.847342 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:55.847318    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:52:56.211719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.211667    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:56.245068 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.245035    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:52:56.782559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.696605    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:52:56.782559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.727981    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:52:56.782559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.767213    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:52:56.782559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.767938    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:52:56.811574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.786282    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:52:56.811574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.786718    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:52:56.836556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.816626    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:52:56.836556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.817017    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:56.897557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.862265    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:52:57.186619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.903707    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:52:57.186619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.910622    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:52:57.241045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:56.972002    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.247725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.247667    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.516692 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.516651    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:52:57.527663 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.527615    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:52:57.775619 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.758987    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:52:57.814103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.814054    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:52:57.815635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.815601    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:52:57.923803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:57.911724    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.136617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.133176    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:52:58.188332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.183895    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:52:58.210662 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.210625    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:52:58.210936 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.210905    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.276560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.254136    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37adea76-6c18-4dc2-95e3-8ba26cb700ce" path="/var/lib/kubelet/pods/37adea76-6c18-4dc2-95e3-8ba26cb700ce/volumes"
Nov 01 08:52:58.276560 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:52:58.254602    4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.118s"
Nov 01 08:52:58.276560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.254707    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" event={"ID":"a35b7a28-8f7d-477f-8584-e999ac50b0dd","Type":"ContainerStarted","Data":"36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61"}
Nov 01 08:52:58.276560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.254761    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:52:58.336661 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.336606    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.410629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.409932    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:52:58.410629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.410001    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:52:58.504090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.504049    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:52:58.504379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.504351    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.687616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.651790    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podStartSLOduration=6.651768416 podStartE2EDuration="6.651768416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:57.925081499 +0000 UTC m=+1251.119476055" watchObservedRunningTime="2026-11-01 08:52:58.651768416 +0000 UTC m=+1251.846162971"
Nov 01 08:52:58.695826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.695694    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Nov 01 08:52:58.759559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.759410    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.829568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.827636    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.829568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.828037    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.870558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.869163    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.912829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:58.912787    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:52:59.011672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.011632    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:52:59.026559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.015615    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.113470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.113428    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:52:59.227574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.220453    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.227947 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.227892    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.338799 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.338662    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.339096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.339067    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:52:59.405695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.405655    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:52:59.406906 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.406869    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:52:59.430570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.403524    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:52:59.777618 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.760287    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Nov 01 08:52:59.777618 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.760840    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:52:59.813212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.813180    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:52:59.836840 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.836799    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:52:59.837340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.837086    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:52:59.840875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.840772    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:52:59.850686 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.850606    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:52:59.856931 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:52:59.856905    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:00.382557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:00.374963    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:53:00.382557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:00.375042    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:53:01.283242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.283199    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"]
Nov 01 08:53:01.284835 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.284805    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a2f83bcf-9efd-4f1f-b59f-9f818a031154" containerName="revoke-keys"
Nov 01 08:53:01.284835 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.284836    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f83bcf-9efd-4f1f-b59f-9f818a031154" containerName="revoke-keys"
Nov 01 08:53:01.285024 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.284976    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f83bcf-9efd-4f1f-b59f-9f818a031154" containerName="revoke-keys"
Nov 01 08:53:01.290308 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.290275    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.314896 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.314866    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.315088 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.314980    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.315088 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.315043    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljns\" (UniqueName: \"kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.319023 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.318996    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"]
Nov 01 08:53:01.383514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.380841    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:53:01.383514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.381021    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:53:01.383514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.381080    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:53:01.383514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.383013    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:53:01.383514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.383067    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:53:01.385222 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.385201    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:53:01.416007 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.415973    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.416148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.416056    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.416148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.416093    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljns\" (UniqueName: \"kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.416791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.416766    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.418383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.418363    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.485772 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.485735    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljns\" (UniqueName: \"kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns\") pod \"maas-api-revoke-sub-93423d91-znnzs\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.604648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.604557    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:01.987982 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:01.987946    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4ba9fd_db71_46d8_b5b8_d0429aab0fd4.slice/crio-54d398d6a5e7de079c816ea56dd75070b60a2dcf4991b111e29b8276882434af WatchSource:0}: Error finding container 54d398d6a5e7de079c816ea56dd75070b60a2dcf4991b111e29b8276882434af: Status 404 returned error can't find the container with id 54d398d6a5e7de079c816ea56dd75070b60a2dcf4991b111e29b8276882434af
Nov 01 08:53:01.988214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:01.988175    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"]
Nov 01 08:53:02.214716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.214686    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"]
Nov 01 08:53:02.219700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.219682    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.226768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.226705    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.226883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.226806    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxvcb\" (UniqueName: \"kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.226984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.226959    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.290334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.290261    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"]
Nov 01 08:53:02.328718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.328666    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.328718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.328719    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxvcb\" (UniqueName: \"kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.328966 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.328821    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.329665 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.329629    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.331678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.331654    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.383497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.383461    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:02.383821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.383797    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:53:02.383973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.383948    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:53:02.386270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.386243    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxvcb\" (UniqueName: \"kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb\") pod \"maas-api-revoke-sub-7175babb-7kdn7\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.532864 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.532834    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:02.554236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.554210    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:02.803896 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.803812    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs" event={"ID":"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4","Type":"ContainerStarted","Data":"685e2234a02604b25878da7f995bb0ed2ca1225fe7bbdf97b1e5416b7551e98f"}
Nov 01 08:53:02.803896 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.803867    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs" event={"ID":"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4","Type":"ContainerStarted","Data":"54d398d6a5e7de079c816ea56dd75070b60a2dcf4991b111e29b8276882434af"}
Nov 01 08:53:02.912717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.912663    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs" podStartSLOduration=1.912649236 podStartE2EDuration="1.912649236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:02.910766911 +0000 UTC m=+1256.105161463" watchObservedRunningTime="2026-11-01 08:53:02.912649236 +0000 UTC m=+1256.107043791"
Nov 01 08:53:02.986185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:02.986148    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"]
Nov 01 08:53:02.986932 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:02.986889    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d368aa_846b_4a0a_9dc7_7aab4d5d7eb6.slice/crio-f8ffd574df681518311dde8d4f48d6a49af9ada485f5c750f1692cb86202a3b1 WatchSource:0}: Error finding container f8ffd574df681518311dde8d4f48d6a49af9ada485f5c750f1692cb86202a3b1: Status 404 returned error can't find the container with id f8ffd574df681518311dde8d4f48d6a49af9ada485f5c750f1692cb86202a3b1
Nov 01 08:53:03.383711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:03.383677    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:53:03.384135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:03.383978    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:53:03.812177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:03.812142    4314 generic.go:356] "Generic (PLEG): container finished" podID="8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" containerID="685e2234a02604b25878da7f995bb0ed2ca1225fe7bbdf97b1e5416b7551e98f" exitCode=0
Nov 01 08:53:03.812383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:03.812227    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs" event={"ID":"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4","Type":"ContainerDied","Data":"685e2234a02604b25878da7f995bb0ed2ca1225fe7bbdf97b1e5416b7551e98f"}
Nov 01 08:53:03.813847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:03.813825    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7" event={"ID":"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6","Type":"ContainerStarted","Data":"12e6e157d509dc66b66a01b094cd96de1aa22c30ace713aebbfdf65874166dcb"}
Nov 01 08:53:03.813932 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:03.813867    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7" event={"ID":"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6","Type":"ContainerStarted","Data":"f8ffd574df681518311dde8d4f48d6a49af9ada485f5c750f1692cb86202a3b1"}
Nov 01 08:53:04.051482 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:04.051431    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7" podStartSLOduration=2.051417 podStartE2EDuration="2.051417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:04.047658714 +0000 UTC m=+1257.242053269" watchObservedRunningTime="2026-11-01 08:53:04.051417 +0000 UTC m=+1257.245811555"
Nov 01 08:53:04.369346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:04.369312    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:53:04.369510 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:04.369364    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:53:04.824391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:04.824360    4314 generic.go:356] "Generic (PLEG): container finished" podID="27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" containerID="12e6e157d509dc66b66a01b094cd96de1aa22c30ace713aebbfdf65874166dcb" exitCode=0
Nov 01 08:53:04.824897 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:04.824442    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7" event={"ID":"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6","Type":"ContainerDied","Data":"12e6e157d509dc66b66a01b094cd96de1aa22c30ace713aebbfdf65874166dcb"}
Nov 01 08:53:04.981853 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:04.981823    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:05.063423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.063390    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns\" (UniqueName: \"kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns\") pod \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") "
Nov 01 08:53:05.063423 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.063431    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca\") pod \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") "
Nov 01 08:53:05.063694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.063632    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca\") pod \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\" (UID: \"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4\") "
Nov 01 08:53:05.063793 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.063764    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca" pod "8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" (UID: "8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:05.064178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.064154    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:05.065666 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.065639    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns" pod "8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" (UID: "8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4"). InnerVolumeSpecName "kube-api-access-5ljns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:05.065773 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.065664    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca" pod "8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" (UID: "8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:05.165870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.165784    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:05.165870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.165811    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ljns\" (UniqueName: \"kubernetes.io/projected/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4-kube-api-access-5ljns\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:05.369448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.369399    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:53:05.369670 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.369465    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 08:53:05.844314 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.844276    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs" event={"ID":"8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4","Type":"ContainerDied","Data":"54d398d6a5e7de079c816ea56dd75070b60a2dcf4991b111e29b8276882434af"}
Nov 01 08:53:05.844780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.844338    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d398d6a5e7de079c816ea56dd75070b60a2dcf4991b111e29b8276882434af"
Nov 01 08:53:05.844780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:05.844455    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"
Nov 01 08:53:06.020300 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.020273    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:06.080109 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.080082    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb\" (UniqueName: \"kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb\") pod \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") "
Nov 01 08:53:06.080268 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.080127    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca\") pod \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") "
Nov 01 08:53:06.080268 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.080239    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca\") pod \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\" (UID: \"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6\") "
Nov 01 08:53:06.080581 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.080558    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca" pod "27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" (UID: "27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:06.080939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.080918    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:06.082243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.082210    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb" pod "27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" (UID: "27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6"). InnerVolumeSpecName "kube-api-access-kxvcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:06.082342 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.082289    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca" pod "27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" (UID: "27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:06.182131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.182053    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:06.182131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.182080    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxvcb\" (UniqueName: \"kubernetes.io/projected/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6-kube-api-access-kxvcb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:06.374258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.374221    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:53:06.374685 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.374662    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:53:06.375616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.375592    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 08:53:06.852794 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.852763    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"
Nov 01 08:53:06.853304 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.852802    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7" event={"ID":"27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6","Type":"ContainerDied","Data":"f8ffd574df681518311dde8d4f48d6a49af9ada485f5c750f1692cb86202a3b1"}
Nov 01 08:53:06.853304 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.852844    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8ffd574df681518311dde8d4f48d6a49af9ada485f5c750f1692cb86202a3b1"
Nov 01 08:53:06.867241 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:06.865610    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:07.137276 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137195    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"]
Nov 01 08:53:07.137807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137788    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" containerName="revoke-keys"
Nov 01 08:53:07.137875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137810    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" containerName="revoke-keys"
Nov 01 08:53:07.137875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137833    4314 cpu_manager.go:414] "containerMap: removing container" podUID="27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" containerName="revoke-keys"
Nov 01 08:53:07.137875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137838    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" containerName="revoke-keys"
Nov 01 08:53:07.137992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137954    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" containerName="revoke-keys"
Nov 01 08:53:07.137992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.137965    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" containerName="revoke-keys"
Nov 01 08:53:07.141889 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.141874    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.191906 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.191872    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:53:07.192070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.191880    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:53:07.192070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.191881    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:53:07.192070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.191979    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:53:07.195891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.195859    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.196334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.196284    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.196434 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.196351    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djz9\" (UniqueName: \"kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.207314 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.207288    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"]
Nov 01 08:53:07.291514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.291471    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:07.297981 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.297955    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.298150 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.298134    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.298195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.298159    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djz9\" (UniqueName: \"kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.298589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.298570    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.300568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.300546    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.355362 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.355328    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djz9\" (UniqueName: \"kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9\") pod \"maas-api-revoke-sub-efa904f3-7hzt5\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.456643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.456524    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:07.838368 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.838334    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"]
Nov 01 08:53:07.838897 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:07.838869    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d098af_f183_4003_9f1f_481b29f10257.slice/crio-01c9330081514809a907e65ccce266acf2b73a92126bb8a8cba4d21ea9d7b769 WatchSource:0}: Error finding container 01c9330081514809a907e65ccce266acf2b73a92126bb8a8cba4d21ea9d7b769: Status 404 returned error can't find the container with id 01c9330081514809a907e65ccce266acf2b73a92126bb8a8cba4d21ea9d7b769
Nov 01 08:53:07.861588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:07.861556    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5" event={"ID":"f5d098af-f183-4003-9f1f-481b29f10257","Type":"ContainerStarted","Data":"01c9330081514809a907e65ccce266acf2b73a92126bb8a8cba4d21ea9d7b769"}
Nov 01 08:53:08.170066 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.170030    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"]
Nov 01 08:53:08.176816 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.176794    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211341 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211316    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211467 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211348    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211467 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211380    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211503    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211622 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211583    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211669    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211691    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcr84\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211735    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.211847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.211828    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.282614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.282579    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"]
Nov 01 08:53:08.313216 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313183    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313226    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313280    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313297    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcr84\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313357 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313348    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313377    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313434    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313460    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313588 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313507    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313789 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313650    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313789 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313757    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313798    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.313984 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.313962    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.314088 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.314052    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.315679 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.315655    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.315909 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.315892    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.371296 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.371265    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.371428 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.371410    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcr84\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.491814 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.491785    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:08.869642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.869610    4314 generic.go:356] "Generic (PLEG): container finished" podID="f5d098af-f183-4003-9f1f-481b29f10257" containerID="c952349ba813f8b3161bba42313ba4279be10cfc36fedcecb615bc6b5332280a" exitCode=0
Nov 01 08:53:08.870075 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.869694    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5" event={"ID":"f5d098af-f183-4003-9f1f-481b29f10257","Type":"ContainerDied","Data":"c952349ba813f8b3161bba42313ba4279be10cfc36fedcecb615bc6b5332280a"}
Nov 01 08:53:08.914601 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:08.914574    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb029da95_7787_4a8e_93af_e74a100f7fb3.slice/crio-05dfd20d1243b2429d44510d333f5742338a1799685c95be288dac0b5d1e7375 WatchSource:0}: Error finding container 05dfd20d1243b2429d44510d333f5742338a1799685c95be288dac0b5d1e7375: Status 404 returned error can't find the container with id 05dfd20d1243b2429d44510d333f5742338a1799685c95be288dac0b5d1e7375
Nov 01 08:53:08.914601 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.914590    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"]
Nov 01 08:53:08.917122 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.917090    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:53:08.917229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.917162    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:53:08.917229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:08.917206    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:53:09.888836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:09.888802    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" event={"ID":"b029da95-7787-4a8e-93af-e74a100f7fb3","Type":"ContainerStarted","Data":"e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5"}
Nov 01 08:53:09.889307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:09.888863    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" event={"ID":"b029da95-7787-4a8e-93af-e74a100f7fb3","Type":"ContainerStarted","Data":"05dfd20d1243b2429d44510d333f5742338a1799685c95be288dac0b5d1e7375"}
Nov 01 08:53:10.054612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.054565    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podStartSLOduration=2.05454968 podStartE2EDuration="2.05454968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:10.050158603 +0000 UTC m=+1263.244553178" watchObservedRunningTime="2026-11-01 08:53:10.05454968 +0000 UTC m=+1263.248944233"
Nov 01 08:53:10.055141 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.055124    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:10.135702 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.135667    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca\") pod \"f5d098af-f183-4003-9f1f-481b29f10257\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") "
Nov 01 08:53:10.135900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.135736    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9\" (UniqueName: \"kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9\") pod \"f5d098af-f183-4003-9f1f-481b29f10257\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") "
Nov 01 08:53:10.135900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.135788    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca\") pod \"f5d098af-f183-4003-9f1f-481b29f10257\" (UID: \"f5d098af-f183-4003-9f1f-481b29f10257\") "
Nov 01 08:53:10.136120 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.136094    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca" pod "f5d098af-f183-4003-9f1f-481b29f10257" (UID: "f5d098af-f183-4003-9f1f-481b29f10257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:10.136346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.136329    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:10.137939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.137911    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca" pod "f5d098af-f183-4003-9f1f-481b29f10257" (UID: "f5d098af-f183-4003-9f1f-481b29f10257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:10.138028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.137989    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9" pod "f5d098af-f183-4003-9f1f-481b29f10257" (UID: "f5d098af-f183-4003-9f1f-481b29f10257"). InnerVolumeSpecName "kube-api-access-4djz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:10.237983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.237949    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4djz9\" (UniqueName: \"kubernetes.io/projected/f5d098af-f183-4003-9f1f-481b29f10257-kube-api-access-4djz9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:10.237983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.237979    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d098af-f183-4003-9f1f-481b29f10257-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:10.492640 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.492603    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:10.494140 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.494110    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:10.494272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.494156    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:10.897173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.897094    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"
Nov 01 08:53:10.897173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.897117    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5" event={"ID":"f5d098af-f183-4003-9f1f-481b29f10257","Type":"ContainerDied","Data":"01c9330081514809a907e65ccce266acf2b73a92126bb8a8cba4d21ea9d7b769"}
Nov 01 08:53:10.897173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:10.897170    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c9330081514809a907e65ccce266acf2b73a92126bb8a8cba4d21ea9d7b769"
Nov 01 08:53:11.325625 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:53:11.325584    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:53:11.493098 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:11.493065    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:11.493248 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:11.493118    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:12.235576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:12.235508    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:12.493188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:12.493099    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:12.493188 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:12.493155    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:13.054824 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:13.054779    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"]
Nov 01 08:53:13.100832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:13.100800    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa83ef41-dqdnk"]
Nov 01 08:53:13.492749 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:13.492715    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:13.493134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:13.492769    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:14.492555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:14.492479    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:14.492815 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:14.492602    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:14.944600 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:14.941886    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80" path="/var/lib/kubelet/pods/fb9fe2fb-6be6-4790-b9a9-8e0aa7c24b80/volumes"
Nov 01 08:53:15.493558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:15.493202    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:15.493558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:15.493268    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:16.493569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:16.493024    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:53:16.493569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:16.493115    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 01 08:53:16.833616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:16.832624    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:16.837553 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:16.833939    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:53:16.914556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:16.913648    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:16.946566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:16.945296    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 01 08:53:17.154613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.150964    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:53:17.178565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.177708    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.259558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.249362    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.422616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.409733    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:53:17.835559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.832789    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.835559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.833555    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:17.846559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.846473    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:53:17.847561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:17.846903    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:53:18.126613 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.125901    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:18.230557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.214673    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:53:18.410621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.410006    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.435556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.218764    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:53:18.435556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.434781    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:53:18.627927 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.561701    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:53:18.627927 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.561771    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.683371 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.683268    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Nov 01 08:53:18.803605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.796036    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.836441 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.835649    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.838333 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.838290    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.851556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.838581    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:53:18.851556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.848067    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.947654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:18.926650    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:19.124174 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.123899    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:19.217862 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.217767    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.293910    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.295391    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.333680    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.333748    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.333798    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.338995    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.357370    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:19.374556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.357761    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:53:19.384983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:19.384952    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:53:23.313224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.313184    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:23.432881 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.432846    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"]
Nov 01 08:53:23.433487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.433472    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f5d098af-f183-4003-9f1f-481b29f10257" containerName="revoke-keys"
Nov 01 08:53:23.433561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.433489    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d098af-f183-4003-9f1f-481b29f10257" containerName="revoke-keys"
Nov 01 08:53:23.433602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.433596    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d098af-f183-4003-9f1f-481b29f10257" containerName="revoke-keys"
Nov 01 08:53:23.438645 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.438621    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.491486 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.491451    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"]
Nov 01 08:53:23.508030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.507998    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.508207 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.508145    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncdg\" (UniqueName: \"kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.508317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.508289    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.532381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.532353    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:53:23.532381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.532360    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:53:23.532381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.532365    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:53:23.533196 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.533181    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:53:23.609388 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.609293    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.609604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.609409    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncdg\" (UniqueName: \"kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.609604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.609477    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.610134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.610114    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.611684 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.611663    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.666836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.666800    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncdg\" (UniqueName: \"kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg\") pod \"maas-api-revoke-sub-3af00103-txhqr\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:23.756988 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:23.756946    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:24.135386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:24.135353    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"]
Nov 01 08:53:24.135515 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:24.135431    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38d11b1_bfd0_45a9_ad5e_be220f986cab.slice/crio-7ccc9a42b26ceb932d56a3942a35656695da803ac5fad745c6247bb07328264d WatchSource:0}: Error finding container 7ccc9a42b26ceb932d56a3942a35656695da803ac5fad745c6247bb07328264d: Status 404 returned error can't find the container with id 7ccc9a42b26ceb932d56a3942a35656695da803ac5fad745c6247bb07328264d
Nov 01 08:53:24.482726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:24.482692    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr" event={"ID":"a38d11b1-bfd0-45a9-ad5e-be220f986cab","Type":"ContainerStarted","Data":"e0a7295743b56fc35eac9826ecabd1dccc7536f6c4268f3d63018ac9c8970228"}
Nov 01 08:53:24.483092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:24.482747    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr" event={"ID":"a38d11b1-bfd0-45a9-ad5e-be220f986cab","Type":"ContainerStarted","Data":"7ccc9a42b26ceb932d56a3942a35656695da803ac5fad745c6247bb07328264d"}
Nov 01 08:53:24.592835 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:24.592779    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr" podStartSLOduration=1.5927644760000002 podStartE2EDuration="1.592764476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:24.591092512 +0000 UTC m=+1277.785487068" watchObservedRunningTime="2026-11-01 08:53:24.592764476 +0000 UTC m=+1277.787159031"
Nov 01 08:53:25.497760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.497724    4314 generic.go:356] "Generic (PLEG): container finished" podID="a38d11b1-bfd0-45a9-ad5e-be220f986cab" containerID="e0a7295743b56fc35eac9826ecabd1dccc7536f6c4268f3d63018ac9c8970228" exitCode=0
Nov 01 08:53:25.498164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.497769    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr" event={"ID":"a38d11b1-bfd0-45a9-ad5e-be220f986cab","Type":"ContainerDied","Data":"e0a7295743b56fc35eac9826ecabd1dccc7536f6c4268f3d63018ac9c8970228"}
Nov 01 08:53:25.658637 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.658594    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"]
Nov 01 08:53:25.662936 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.662912    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.721033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.717470    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"]
Nov 01 08:53:25.739870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.739830    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqf8p\" (UniqueName: \"kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.740061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.739908    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.740061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.739963    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.841950 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.841848    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf8p\" (UniqueName: \"kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.842138 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.841951    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.842138 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.842027    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.842827 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.842804    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.844396 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.844378    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.899710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.899682    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqf8p\" (UniqueName: \"kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p\") pod \"maas-api-revoke-sub-4797ed5c-md8qv\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:25.976508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:25.976470    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:26.347177 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:26.347140    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb98d16_d7f3_48c6_af69_fa394ec87309.slice/crio-ec932445a4c3492c997297a514fc03f6b44d5a003f8b21ffaa201ff152084672 WatchSource:0}: Error finding container ec932445a4c3492c997297a514fc03f6b44d5a003f8b21ffaa201ff152084672: Status 404 returned error can't find the container with id ec932445a4c3492c997297a514fc03f6b44d5a003f8b21ffaa201ff152084672
Nov 01 08:53:26.348491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.348464    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"]
Nov 01 08:53:26.506448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.506408    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv" event={"ID":"afb98d16-d7f3-48c6-af69-fa394ec87309","Type":"ContainerStarted","Data":"ec932445a4c3492c997297a514fc03f6b44d5a003f8b21ffaa201ff152084672"}
Nov 01 08:53:26.646310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.646287    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:26.754139 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.754106    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca\") pod \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") "
Nov 01 08:53:26.754324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.754221    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca\") pod \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") "
Nov 01 08:53:26.754407 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.754330    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg\" (UniqueName: \"kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg\") pod \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\" (UID: \"a38d11b1-bfd0-45a9-ad5e-be220f986cab\") "
Nov 01 08:53:26.754501 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.754475    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca" pod "a38d11b1-bfd0-45a9-ad5e-be220f986cab" (UID: "a38d11b1-bfd0-45a9-ad5e-be220f986cab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:26.754830 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.754811    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:26.756807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.756675    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca" pod "a38d11b1-bfd0-45a9-ad5e-be220f986cab" (UID: "a38d11b1-bfd0-45a9-ad5e-be220f986cab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:26.757016 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.756991    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg" pod "a38d11b1-bfd0-45a9-ad5e-be220f986cab" (UID: "a38d11b1-bfd0-45a9-ad5e-be220f986cab"). InnerVolumeSpecName "kube-api-access-7ncdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:26.856161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.856128    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d11b1-bfd0-45a9-ad5e-be220f986cab-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:26.856161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:26.856161    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncdg\" (UniqueName: \"kubernetes.io/projected/a38d11b1-bfd0-45a9-ad5e-be220f986cab-kube-api-access-7ncdg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:27.514751 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:27.514669    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"
Nov 01 08:53:27.514751 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:27.514692    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr" event={"ID":"a38d11b1-bfd0-45a9-ad5e-be220f986cab","Type":"ContainerDied","Data":"7ccc9a42b26ceb932d56a3942a35656695da803ac5fad745c6247bb07328264d"}
Nov 01 08:53:27.514751 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:27.514744    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ccc9a42b26ceb932d56a3942a35656695da803ac5fad745c6247bb07328264d"
Nov 01 08:53:27.517012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:27.516677    4314 generic.go:356] "Generic (PLEG): container finished" podID="afb98d16-d7f3-48c6-af69-fa394ec87309" containerID="3c7322f55113663329370ea216cf9ebca3413113126c3cfbc742c8b18769fdde" exitCode=0
Nov 01 08:53:27.517012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:27.516762    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv" event={"ID":"afb98d16-d7f3-48c6-af69-fa394ec87309","Type":"ContainerDied","Data":"3c7322f55113663329370ea216cf9ebca3413113126c3cfbc742c8b18769fdde"}
Nov 01 08:53:28.271090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.271049    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:28.671401 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.671375    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:28.780049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.779780    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca\") pod \"afb98d16-d7f3-48c6-af69-fa394ec87309\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") "
Nov 01 08:53:28.780049 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.779975    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p\" (UniqueName: \"kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p\") pod \"afb98d16-d7f3-48c6-af69-fa394ec87309\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") "
Nov 01 08:53:28.780903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.780198    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca\") pod \"afb98d16-d7f3-48c6-af69-fa394ec87309\" (UID: \"afb98d16-d7f3-48c6-af69-fa394ec87309\") "
Nov 01 08:53:28.780903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.780739    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca" pod "afb98d16-d7f3-48c6-af69-fa394ec87309" (UID: "afb98d16-d7f3-48c6-af69-fa394ec87309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:28.782419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.782380    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca" pod "afb98d16-d7f3-48c6-af69-fa394ec87309" (UID: "afb98d16-d7f3-48c6-af69-fa394ec87309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:28.782999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.782974    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p" pod "afb98d16-d7f3-48c6-af69-fa394ec87309" (UID: "afb98d16-d7f3-48c6-af69-fa394ec87309"). InnerVolumeSpecName "kube-api-access-hqf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:28.882435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.882397    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:28.882435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.882429    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb98d16-d7f3-48c6-af69-fa394ec87309-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:28.882435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:28.882439    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqf8p\" (UniqueName: \"kubernetes.io/projected/afb98d16-d7f3-48c6-af69-fa394ec87309-kube-api-access-hqf8p\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:29.530838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:29.530806    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"
Nov 01 08:53:29.531004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:29.530838    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv" event={"ID":"afb98d16-d7f3-48c6-af69-fa394ec87309","Type":"ContainerDied","Data":"ec932445a4c3492c997297a514fc03f6b44d5a003f8b21ffaa201ff152084672"}
Nov 01 08:53:29.531004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:29.530889    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec932445a4c3492c997297a514fc03f6b44d5a003f8b21ffaa201ff152084672"
Nov 01 08:53:30.046574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:30.046525    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"]
Nov 01 08:53:30.050946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:30.050918    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5de02b11-rdpmd"]
Nov 01 08:53:30.950554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:30.950325    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f77e973-7883-42c4-8395-e97af1530cc0" path="/var/lib/kubelet/pods/9f77e973-7883-42c4-8395-e97af1530cc0/volumes"
Nov 01 08:53:32.120636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.099589    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"]
Nov 01 08:53:32.156558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.150276    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f536057-j7qsd"]
Nov 01 08:53:32.342653 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.342606    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.348937 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.348900    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:53:32.355851 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.349601    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:53:32.368663 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.368617    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.404620 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.402629    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.534978 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.534919    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.941568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.940645    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.966558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:32.964800    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5789434-7044-43fe-be5f-4a2a2784300d" path="/var/lib/kubelet/pods/a5789434-7044-43fe-be5f-4a2a2784300d/volumes"
Nov 01 08:53:33.131688 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.131636    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.357558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.354563    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:53:33.398612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.357818    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.403589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.399563    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:53:33.450714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.450677    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:53:33.478546 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.361605    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.486557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.408495    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.508558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.491068    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:33.508893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.508865    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:53:33.509759 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.371655    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:53:33.605987 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.605952    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:53:33.952556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.951669    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.956554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:33.953213    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:53:34.187556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.162852    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:34.255043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.255017    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:53:34.256113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.244970    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:34.412674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.412623    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:34.436200 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.428662    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:34.502201 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.501567    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:53:34.503045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.502461    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:53:34.612605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.611627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:53:34.958612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:34.958069    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.386057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.386003    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.415822 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.415776    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:53:35.431656 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.431613    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.447100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.447063    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:53:35.447338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.447307    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.517674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.517620    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.518242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.518211    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:53:35.613007 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.612661    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:53:35.999770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:35.999727    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:36.151726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.150438    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:53:36.164803 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.164506    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:53:36.186213 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.173709    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:53:36.190563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.190524    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:53:36.228562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.228035    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:53:36.228562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.228344    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:53:36.234198 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.234168    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:36.500164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500134    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"]
Nov 01 08:53:36.500740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500724    4314 cpu_manager.go:414] "containerMap: removing container" podUID="afb98d16-d7f3-48c6-af69-fa394ec87309" containerName="revoke-keys"
Nov 01 08:53:36.500801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500740    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb98d16-d7f3-48c6-af69-fa394ec87309" containerName="revoke-keys"
Nov 01 08:53:36.500801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500773    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a38d11b1-bfd0-45a9-ad5e-be220f986cab" containerName="revoke-keys"
Nov 01 08:53:36.500801 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500779    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38d11b1-bfd0-45a9-ad5e-be220f986cab" containerName="revoke-keys"
Nov 01 08:53:36.500893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500862    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb98d16-d7f3-48c6-af69-fa394ec87309" containerName="revoke-keys"
Nov 01 08:53:36.500893 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.500872    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38d11b1-bfd0-45a9-ad5e-be220f986cab" containerName="revoke-keys"
Nov 01 08:53:36.506362 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.506340    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.555124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.555095    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:53:36.555306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.555135    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:53:36.555306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.555135    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:53:36.555306 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.555177    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:53:36.561053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.561000    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.561489 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.561090    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.561489 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.561201    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qvq\" (UniqueName: \"kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.564448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.564423    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"]
Nov 01 08:53:36.662506 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.662476    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.662699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.662579    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.662699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.662688    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qvq\" (UniqueName: \"kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.663303 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.663282    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.664922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.664902    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.720940 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.720910    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qvq\" (UniqueName: \"kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq\") pod \"maas-api-revoke-sub-003cca03-kkv29\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:36.827025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:36.826953    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:37.203926 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:37.203898    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17124d9e_fb64_45bc_8e07_a804a8e09456.slice/crio-8a6e8d794491a0d001b552efdf054f052dc2248c697affb6ab6c14bd8f884af7 WatchSource:0}: Error finding container 8a6e8d794491a0d001b552efdf054f052dc2248c697affb6ab6c14bd8f884af7: Status 404 returned error can't find the container with id 8a6e8d794491a0d001b552efdf054f052dc2248c697affb6ab6c14bd8f884af7
Nov 01 08:53:37.204877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:37.204854    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"]
Nov 01 08:53:38.018642 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:38.018602    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29" event={"ID":"17124d9e-fb64-45bc-8e07-a804a8e09456","Type":"ContainerStarted","Data":"b3aab14e6ea4f338e40a6e50b4cee8d591db7f762fa68a615abb2e5cdf540213"}
Nov 01 08:53:38.019037 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:38.018665    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29" event={"ID":"17124d9e-fb64-45bc-8e07-a804a8e09456","Type":"ContainerStarted","Data":"8a6e8d794491a0d001b552efdf054f052dc2248c697affb6ab6c14bd8f884af7"}
Nov 01 08:53:38.133687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:38.133628    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29" podStartSLOduration=2.133609061 podStartE2EDuration="2.133609061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:38.126009983 +0000 UTC m=+1291.320404538" watchObservedRunningTime="2026-11-01 08:53:38.133609061 +0000 UTC m=+1291.328003618"
Nov 01 08:53:38.532832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:38.532799    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:39.034392 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.034359    4314 generic.go:356] "Generic (PLEG): container finished" podID="17124d9e-fb64-45bc-8e07-a804a8e09456" containerID="b3aab14e6ea4f338e40a6e50b4cee8d591db7f762fa68a615abb2e5cdf540213" exitCode=0
Nov 01 08:53:39.034826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.034448    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29" event={"ID":"17124d9e-fb64-45bc-8e07-a804a8e09456","Type":"ContainerDied","Data":"b3aab14e6ea4f338e40a6e50b4cee8d591db7f762fa68a615abb2e5cdf540213"}
Nov 01 08:53:39.273287 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.273253    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:39.539136 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.539098    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"]
Nov 01 08:53:39.544542 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.544506    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.594718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.594672    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"kube-root-ca.crt\""
Nov 01 08:53:39.601005 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.600971    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"]
Nov 01 08:53:39.601865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.601836    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.601996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.601913    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.601996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.601968    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdg4\" (UniqueName: \"kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.602116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.602061    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.602173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.602115    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.602173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.602161    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.627182 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.627154    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:39.636626 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.636603    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"openshift-service-ca.crt\""
Nov 01 08:53:39.637768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.637748    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"distinct-w3-54e01b-kserve-self-signed-certs\""
Nov 01 08:53:39.637854 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.637753    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"default-dockercfg-fp8xg\""
Nov 01 08:53:39.703696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.703657    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.703900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.703749    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.703900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.703831    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdg4\" (UniqueName: \"kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.704027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.703973    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.704077 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.704041    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.704129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.704098    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.704270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.704245    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.704445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.704422    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.704512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.704467    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.706616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.706579    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.706821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.706802    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.765434 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.765402    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdg4\" (UniqueName: \"kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4\") pod \"distinct-w3-54e01b-kserve-78c6db7969-vcfkp\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:39.857899 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:39.857820    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:40.174662 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.174635    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:40.213167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.213136    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca\") pod \"17124d9e-fb64-45bc-8e07-a804a8e09456\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") "
Nov 01 08:53:40.213338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.213273    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq\" (UniqueName: \"kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq\") pod \"17124d9e-fb64-45bc-8e07-a804a8e09456\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") "
Nov 01 08:53:40.213417 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.213340    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca\") pod \"17124d9e-fb64-45bc-8e07-a804a8e09456\" (UID: \"17124d9e-fb64-45bc-8e07-a804a8e09456\") "
Nov 01 08:53:40.213662 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.213637    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca" pod "17124d9e-fb64-45bc-8e07-a804a8e09456" (UID: "17124d9e-fb64-45bc-8e07-a804a8e09456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:40.213755 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.213741    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:40.215270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.215245    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca" pod "17124d9e-fb64-45bc-8e07-a804a8e09456" (UID: "17124d9e-fb64-45bc-8e07-a804a8e09456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:40.215365 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.215294    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq" pod "17124d9e-fb64-45bc-8e07-a804a8e09456" (UID: "17124d9e-fb64-45bc-8e07-a804a8e09456"). InnerVolumeSpecName "kube-api-access-n8qvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:40.264690 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.264667    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"]
Nov 01 08:53:40.264991 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:40.264973    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b5dc9ea_7340_4c0a_8f29_daac7de1c4ab.slice/crio-1bf34ce80ebe5506e03f1eb0c4cdc4805faa9bb665cb5adb25cacc54d5f97b34 WatchSource:0}: Error finding container 1bf34ce80ebe5506e03f1eb0c4cdc4805faa9bb665cb5adb25cacc54d5f97b34: Status 404 returned error can't find the container with id 1bf34ce80ebe5506e03f1eb0c4cdc4805faa9bb665cb5adb25cacc54d5f97b34
Nov 01 08:53:40.314954 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.314925    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qvq\" (UniqueName: \"kubernetes.io/projected/17124d9e-fb64-45bc-8e07-a804a8e09456-kube-api-access-n8qvq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:40.314954 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:40.314951    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17124d9e-fb64-45bc-8e07-a804a8e09456-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:41.049668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.049629    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" event={"ID":"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab","Type":"ContainerStarted","Data":"21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97"}
Nov 01 08:53:41.049967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.049908    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" event={"ID":"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab","Type":"ContainerStarted","Data":"1bf34ce80ebe5506e03f1eb0c4cdc4805faa9bb665cb5adb25cacc54d5f97b34"}
Nov 01 08:53:41.051688 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.051668    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"
Nov 01 08:53:41.051799 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.051712    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29" event={"ID":"17124d9e-fb64-45bc-8e07-a804a8e09456","Type":"ContainerDied","Data":"8a6e8d794491a0d001b552efdf054f052dc2248c697affb6ab6c14bd8f884af7"}
Nov 01 08:53:41.051799 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.051761    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a6e8d794491a0d001b552efdf054f052dc2248c697affb6ab6c14bd8f884af7"
Nov 01 08:53:41.162142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.162086    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podStartSLOduration=2.162071286 podStartE2EDuration="2.162071286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:41.158447964 +0000 UTC m=+1294.352842520" watchObservedRunningTime="2026-11-01 08:53:41.162071286 +0000 UTC m=+1294.356465841"
Nov 01 08:53:41.328903 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:53:41.328803    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:53:41.858057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.858025    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:41.923327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.923296    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"]
Nov 01 08:53:41.923899 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.923885    4314 cpu_manager.go:414] "containerMap: removing container" podUID="17124d9e-fb64-45bc-8e07-a804a8e09456" containerName="revoke-keys"
Nov 01 08:53:41.923952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.923900    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17124d9e-fb64-45bc-8e07-a804a8e09456" containerName="revoke-keys"
Nov 01 08:53:41.923996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.923987    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="17124d9e-fb64-45bc-8e07-a804a8e09456" containerName="revoke-keys"
Nov 01 08:53:41.928073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.928053    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:41.979889 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.979858    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:53:41.982052 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:41.982028    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"]
Nov 01 08:53:42.022422 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.022361    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:53:42.022610 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.022490    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:53:42.036258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.036231    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.036356 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.036322    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.036404 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.036392    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbkk\" (UniqueName: \"kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.080046 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.080024    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:53:42.138198 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.138111    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.138198 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.138169    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.138405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.138201    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbkk\" (UniqueName: \"kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.138833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.138812    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.140460 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.140439    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.195987 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.195963    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbkk\" (UniqueName: \"kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk\") pod \"maas-api-revoke-sub-b329ab73-7pfsr\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.241263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.241232    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:42.609394 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:42.609358    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf60a7d_b6b0_442e_bee1_a790ac5f0748.slice/crio-ae18215667595dabb9d097c243ec73e3b84d9b4932bdf062b14685f484153205 WatchSource:0}: Error finding container ae18215667595dabb9d097c243ec73e3b84d9b4932bdf062b14685f484153205: Status 404 returned error can't find the container with id ae18215667595dabb9d097c243ec73e3b84d9b4932bdf062b14685f484153205
Nov 01 08:53:42.609767 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:42.609590    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"]
Nov 01 08:53:43.066617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:43.066578    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr" event={"ID":"eaf60a7d-b6b0-442e-bee1-a790ac5f0748","Type":"ContainerStarted","Data":"c8eb1ee3b9513c2a194469a4d8fd479473f8dc74e69ae214228e90cad6e89b75"}
Nov 01 08:53:43.066809 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:43.066632    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr" event={"ID":"eaf60a7d-b6b0-442e-bee1-a790ac5f0748","Type":"ContainerStarted","Data":"ae18215667595dabb9d097c243ec73e3b84d9b4932bdf062b14685f484153205"}
Nov 01 08:53:43.178040 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:43.177937    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr" podStartSLOduration=2.17792446 podStartE2EDuration="2.17792446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:43.174391482 +0000 UTC m=+1296.368786037" watchObservedRunningTime="2026-11-01 08:53:43.17792446 +0000 UTC m=+1296.372319044"
Nov 01 08:53:44.074739 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:44.074697    4314 generic.go:356] "Generic (PLEG): container finished" podID="eaf60a7d-b6b0-442e-bee1-a790ac5f0748" containerID="c8eb1ee3b9513c2a194469a4d8fd479473f8dc74e69ae214228e90cad6e89b75" exitCode=0
Nov 01 08:53:44.075153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:44.074783    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr" event={"ID":"eaf60a7d-b6b0-442e-bee1-a790ac5f0748","Type":"ContainerDied","Data":"c8eb1ee3b9513c2a194469a4d8fd479473f8dc74e69ae214228e90cad6e89b75"}
Nov 01 08:53:44.183281 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:44.183249    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:45.260724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.260699    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:45.382787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.382444    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca\") pod \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") "
Nov 01 08:53:45.382787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.382524    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca\") pod \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") "
Nov 01 08:53:45.383131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.382831    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk\" (UniqueName: \"kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk\") pod \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\" (UID: \"eaf60a7d-b6b0-442e-bee1-a790ac5f0748\") "
Nov 01 08:53:45.383131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.382875    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca" pod "eaf60a7d-b6b0-442e-bee1-a790ac5f0748" (UID: "eaf60a7d-b6b0-442e-bee1-a790ac5f0748"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:45.383338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.383317    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:45.386835 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.386767    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk" pod "eaf60a7d-b6b0-442e-bee1-a790ac5f0748" (UID: "eaf60a7d-b6b0-442e-bee1-a790ac5f0748"). InnerVolumeSpecName "kube-api-access-wsbkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:45.386959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.386868    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca" pod "eaf60a7d-b6b0-442e-bee1-a790ac5f0748" (UID: "eaf60a7d-b6b0-442e-bee1-a790ac5f0748"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:45.484171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.484135    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbkk\" (UniqueName: \"kubernetes.io/projected/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-kube-api-access-wsbkk\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:45.484171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:45.484164    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf60a7d-b6b0-442e-bee1-a790ac5f0748-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:46.099000 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:46.098973    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"
Nov 01 08:53:46.099170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:46.098963    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr" event={"ID":"eaf60a7d-b6b0-442e-bee1-a790ac5f0748","Type":"ContainerDied","Data":"ae18215667595dabb9d097c243ec73e3b84d9b4932bdf062b14685f484153205"}
Nov 01 08:53:46.099170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:46.099108    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae18215667595dabb9d097c243ec73e3b84d9b4932bdf062b14685f484153205"
Nov 01 08:53:47.058643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:47.058601    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"]
Nov 01 08:53:47.103609 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:47.103569    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996da403-lfcdd"]
Nov 01 08:53:48.930036 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:48.929939    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83eaa36f-a4a4-4571-89e6-0873c9a602eb" path="/var/lib/kubelet/pods/83eaa36f-a4a4-4571-89e6-0873c9a602eb/volumes"
Nov 01 08:53:49.032151 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:49.032093    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:49.858437 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:49.858398    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:49.871377 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:49.871345    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:50.142671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:50.142600    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:52.504557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.503627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:52.534559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.533626    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": context deadline exceeded"
Nov 01 08:53:52.628559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.628101    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:52.790616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.777820    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"]
Nov 01 08:53:52.795559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.793793    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eaf60a7d-b6b0-442e-bee1-a790ac5f0748" containerName="revoke-keys"
Nov 01 08:53:52.795559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.793825    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf60a7d-b6b0-442e-bee1-a790ac5f0748" containerName="revoke-keys"
Nov 01 08:53:52.795559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.793997    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf60a7d-b6b0-442e-bee1-a790ac5f0748" containerName="revoke-keys"
Nov 01 08:53:52.808554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.806821    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:52.883562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.882900    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:53:52.883562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.883054    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:53:52.883562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.883184    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:53:52.907787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.907731    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:53:52.930556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:52.920238    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"]
Nov 01 08:53:53.062615 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.038420    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:53.080192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.061713    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.198560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.146664    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.198560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.179041    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.201364 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.200339    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.255822 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.236614    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.283599 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.278082    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.283924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.283899    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgljz\" (UniqueName: \"kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.412623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.379434    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"]
Nov 01 08:53:53.412623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.387351    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgljz\" (UniqueName: \"kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.412623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.402709    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.412623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.403080    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.412623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.403961    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.412623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.410192    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2769b8e-vwrns"]
Nov 01 08:53:53.450328 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.450292    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.450668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.450632    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.502339 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.502266    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgljz\" (UniqueName: \"kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz\") pod \"maas-api-revoke-sub-511c8177-mptbg\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:53.512587 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.506619    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.512587 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.509246    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:53:53.640558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.640037    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.663612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.659634    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:53:53.788557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:53.785596    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:54.081699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.081628    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:54.108557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.083984    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:53:54.207558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.203635    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:53:54.224777 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.224663    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:54.226669 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.226639    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:53:54.226894 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.226878    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:53:54.292740 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.277127    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:53:54.345059 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.294329    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 08:53:54.373556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.349006    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:54.373910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.373883    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:54.529677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.529626    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:53:55.334294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.334250    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:55.341232 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.557101    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:55.346555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:54.943024    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:55.433687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.346804    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:53:55.434191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.354350    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:53:55.629930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.376508    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:55.629930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.376577    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:53:55.629930 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.463512    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:53:55.847850 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:55.821998    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:56.277056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.276471    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e198ca46-d32e-4606-99fc-45e1070d5364" path="/var/lib/kubelet/pods/e198ca46-d32e-4606-99fc-45e1070d5364/volumes"
Nov 01 08:53:56.277056 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.276985    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"]
Nov 01 08:53:56.299179 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.299152    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:53:56.338064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.338021    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:56.356564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.350606    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:56.361560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.360136    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:53:56.411557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.401996    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:53:56.411557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.402021    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:53:56.411557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.402065    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:56.411557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.402623    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:53:56.415324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.415300    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:53:56.417227 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.417206    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:53:56.421340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.420281    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:53:56.422219 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.422194    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg" event={"ID":"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e","Type":"ContainerStarted","Data":"7bfb4ea5e2ec174c20762bd24fa13dd072bc99fc45bee0d2b8282340865f597f"}
Nov 01 08:53:56.425358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:56.425341    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:53:57.431269 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:57.431178    4314 generic.go:356] "Generic (PLEG): container finished" podID="afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" containerID="6eea4118fe453fda435ae7cdeed0ffde81cfca8a2cefcf640f46b99aaa70b5f6" exitCode=0
Nov 01 08:53:57.431727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:57.431269    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg" event={"ID":"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e","Type":"ContainerDied","Data":"6eea4118fe453fda435ae7cdeed0ffde81cfca8a2cefcf640f46b99aaa70b5f6"}
Nov 01 08:53:58.002472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.002440    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"]
Nov 01 08:53:58.006658 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.006637    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.061129 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.061103    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"]
Nov 01 08:53:58.126443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.126409    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.126629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.126608    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.126697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.126642    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrdg\" (UniqueName: \"kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.228134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.228100    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.228285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.228141    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrdg\" (UniqueName: \"kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.228285 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.228242    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.228888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.228863    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.230522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.230504    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.285742 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.285685    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrdg\" (UniqueName: \"kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg\") pod \"maas-api-revoke-sub-717ab89d-j6grf\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.320595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.320566    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:53:58.589409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.589385    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:58.633663 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.633627    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca\") pod \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") "
Nov 01 08:53:58.633808 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.633767    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz\" (UniqueName: \"kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz\") pod \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") "
Nov 01 08:53:58.633876 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.633819    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca\") pod \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\" (UID: \"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e\") "
Nov 01 08:53:58.634205 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.634175    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca" pod "afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" (UID: "afa149c0-c9a7-4f8b-b2a2-b42c264ba41e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:53:58.634578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.634554    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:58.636050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.636020    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca" pod "afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" (UID: "afa149c0-c9a7-4f8b-b2a2-b42c264ba41e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:58.636168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.636150    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz" pod "afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" (UID: "afa149c0-c9a7-4f8b-b2a2-b42c264ba41e"). InnerVolumeSpecName "kube-api-access-bgljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:58.694647 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.694620    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"]
Nov 01 08:53:58.694796 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:53:58.694773    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b8240b_c0c8_4d14_8421_5a84f1caea3d.slice/crio-2979be0c14f4cce39ec7bc036d7c8a6f437dc948f0b763666db0ca6339a5bc82 WatchSource:0}: Error finding container 2979be0c14f4cce39ec7bc036d7c8a6f437dc948f0b763666db0ca6339a5bc82: Status 404 returned error can't find the container with id 2979be0c14f4cce39ec7bc036d7c8a6f437dc948f0b763666db0ca6339a5bc82
Nov 01 08:53:58.736053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.736030    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:58.736134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:58.736055    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgljz\" (UniqueName: \"kubernetes.io/projected/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e-kube-api-access-bgljz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:59.457127 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:59.457090    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf" event={"ID":"d8b8240b-c0c8-4d14-8421-5a84f1caea3d","Type":"ContainerStarted","Data":"d8b980ce40672354ca5a3223a1ca50c1f3a4aa999e6a09790b12e3467172aedc"}
Nov 01 08:53:59.457302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:59.457178    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf" event={"ID":"d8b8240b-c0c8-4d14-8421-5a84f1caea3d","Type":"ContainerStarted","Data":"2979be0c14f4cce39ec7bc036d7c8a6f437dc948f0b763666db0ca6339a5bc82"}
Nov 01 08:53:59.458899 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:59.458871    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"
Nov 01 08:53:59.458999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:59.458905    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg" event={"ID":"afa149c0-c9a7-4f8b-b2a2-b42c264ba41e","Type":"ContainerDied","Data":"7bfb4ea5e2ec174c20762bd24fa13dd072bc99fc45bee0d2b8282340865f597f"}
Nov 01 08:53:59.458999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:59.458948    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bfb4ea5e2ec174c20762bd24fa13dd072bc99fc45bee0d2b8282340865f597f"
Nov 01 08:53:59.567144 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:53:59.567094    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf" podStartSLOduration=2.567080978 podStartE2EDuration="2.567080978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:59.564663078 +0000 UTC m=+1312.759057634" watchObservedRunningTime="2026-11-01 08:53:59.567080978 +0000 UTC m=+1312.761475532"
Nov 01 08:54:00.309703 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.309672    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:00.468608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.468574    4314 generic.go:356] "Generic (PLEG): container finished" podID="d8b8240b-c0c8-4d14-8421-5a84f1caea3d" containerID="d8b980ce40672354ca5a3223a1ca50c1f3a4aa999e6a09790b12e3467172aedc" exitCode=0
Nov 01 08:54:00.468810 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.468655    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf" event={"ID":"d8b8240b-c0c8-4d14-8421-5a84f1caea3d","Type":"ContainerDied","Data":"d8b980ce40672354ca5a3223a1ca50c1f3a4aa999e6a09790b12e3467172aedc"}
Nov 01 08:54:00.627560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.627456    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"]
Nov 01 08:54:00.628952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.628925    4314 cpu_manager.go:414] "containerMap: removing container" podUID="afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" containerName="revoke-keys"
Nov 01 08:54:00.628952 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.628951    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" containerName="revoke-keys"
Nov 01 08:54:00.629100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.629087    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" containerName="revoke-keys"
Nov 01 08:54:00.633089 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.633069    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.689951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.689922    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"]
Nov 01 08:54:00.760194 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.760155    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.760415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.760269    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crm9n\" (UniqueName: \"kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.760415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.760378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.862278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.862240    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crm9n\" (UniqueName: \"kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.862454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.862326    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.862514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.862495    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.863033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.863007    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.864806 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.864782    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.922287 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.922215    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crm9n\" (UniqueName: \"kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n\") pod \"maas-api-revoke-sub-94afd23a-l26kt\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:00.949035 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:00.949007    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:01.322931 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.322903    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"]
Nov 01 08:54:01.323278 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:01.323180    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642b038d_db97_4f6b_b976_191dd2a175ea.slice/crio-73373a24c82c759b362326dcc3c1f0ef7c44a74776c21ebb49ca5873eb323bde WatchSource:0}: Error finding container 73373a24c82c759b362326dcc3c1f0ef7c44a74776c21ebb49ca5873eb323bde: Status 404 returned error can't find the container with id 73373a24c82c759b362326dcc3c1f0ef7c44a74776c21ebb49ca5873eb323bde
Nov 01 08:54:01.477754 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.477725    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt" event={"ID":"642b038d-db97-4f6b-b976-191dd2a175ea","Type":"ContainerStarted","Data":"73373a24c82c759b362326dcc3c1f0ef7c44a74776c21ebb49ca5873eb323bde"}
Nov 01 08:54:01.620850 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.620826    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:54:01.673089 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.673065    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca\") pod \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") "
Nov 01 08:54:01.673218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.673134    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca\") pod \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") "
Nov 01 08:54:01.673282 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.673266    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg\" (UniqueName: \"kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg\") pod \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\" (UID: \"d8b8240b-c0c8-4d14-8421-5a84f1caea3d\") "
Nov 01 08:54:01.673439 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.673420    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca" pod "d8b8240b-c0c8-4d14-8421-5a84f1caea3d" (UID: "d8b8240b-c0c8-4d14-8421-5a84f1caea3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:01.673832 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.673814    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:01.675255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.675223    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca" pod "d8b8240b-c0c8-4d14-8421-5a84f1caea3d" (UID: "d8b8240b-c0c8-4d14-8421-5a84f1caea3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:01.675404 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.675381    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg" pod "d8b8240b-c0c8-4d14-8421-5a84f1caea3d" (UID: "d8b8240b-c0c8-4d14-8421-5a84f1caea3d"). InnerVolumeSpecName "kube-api-access-rbrdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:01.775176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.775132    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:01.775176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:01.775168    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbrdg\" (UniqueName: \"kubernetes.io/projected/d8b8240b-c0c8-4d14-8421-5a84f1caea3d-kube-api-access-rbrdg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:02.486317 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:02.486285    4314 generic.go:356] "Generic (PLEG): container finished" podID="642b038d-db97-4f6b-b976-191dd2a175ea" containerID="c14486d788db31a9384f3d0306aa31e743cadb586540ed18ff46ad069fc646c9" exitCode=0
Nov 01 08:54:02.486771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:02.486371    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt" event={"ID":"642b038d-db97-4f6b-b976-191dd2a175ea","Type":"ContainerDied","Data":"c14486d788db31a9384f3d0306aa31e743cadb586540ed18ff46ad069fc646c9"}
Nov 01 08:54:02.487769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:02.487750    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"
Nov 01 08:54:02.487872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:02.487779    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf" event={"ID":"d8b8240b-c0c8-4d14-8421-5a84f1caea3d","Type":"ContainerDied","Data":"2979be0c14f4cce39ec7bc036d7c8a6f437dc948f0b763666db0ca6339a5bc82"}
Nov 01 08:54:02.487872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:02.487822    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2979be0c14f4cce39ec7bc036d7c8a6f437dc948f0b763666db0ca6339a5bc82"
Nov 01 08:54:03.659450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.659423    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:03.700038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.699992    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n\" (UniqueName: \"kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n\") pod \"642b038d-db97-4f6b-b976-191dd2a175ea\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") "
Nov 01 08:54:03.700038 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.700052    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca\") pod \"642b038d-db97-4f6b-b976-191dd2a175ea\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") "
Nov 01 08:54:03.700321 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.700083    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca\") pod \"642b038d-db97-4f6b-b976-191dd2a175ea\" (UID: \"642b038d-db97-4f6b-b976-191dd2a175ea\") "
Nov 01 08:54:03.700515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.700490    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca" pod "642b038d-db97-4f6b-b976-191dd2a175ea" (UID: "642b038d-db97-4f6b-b976-191dd2a175ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:03.702193 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.702171    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca" pod "642b038d-db97-4f6b-b976-191dd2a175ea" (UID: "642b038d-db97-4f6b-b976-191dd2a175ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:03.702266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.702188    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n" pod "642b038d-db97-4f6b-b976-191dd2a175ea" (UID: "642b038d-db97-4f6b-b976-191dd2a175ea"). InnerVolumeSpecName "kube-api-access-crm9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:03.802270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.802177    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crm9n\" (UniqueName: \"kubernetes.io/projected/642b038d-db97-4f6b-b976-191dd2a175ea-kube-api-access-crm9n\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:03.802270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.802221    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:03.802270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:03.802240    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642b038d-db97-4f6b-b976-191dd2a175ea-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:04.051997 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:04.051959    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"]
Nov 01 08:54:04.056570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:04.056482    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49527f3f-rc5bn"]
Nov 01 08:54:04.503307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:04.503278    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"
Nov 01 08:54:04.503482 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:04.503303    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt" event={"ID":"642b038d-db97-4f6b-b976-191dd2a175ea","Type":"ContainerDied","Data":"73373a24c82c759b362326dcc3c1f0ef7c44a74776c21ebb49ca5873eb323bde"}
Nov 01 08:54:04.503482 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:04.503350    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73373a24c82c759b362326dcc3c1f0ef7c44a74776c21ebb49ca5873eb323bde"
Nov 01 08:54:04.928809 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:04.928726    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21033d22-a443-489d-a1df-2336b8091dfb" path="/var/lib/kubelet/pods/21033d22-a443-489d-a1df-2336b8091dfb/volumes"
Nov 01 08:54:05.182589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:05.182466    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:07.188164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:07.188115    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:07.265583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:07.265555    4314 scope.go:122] "RemoveContainer" containerID="d0787ff83585bd7ea2f9d134ec964022504a5bd7bc9f8964f73188877b4b51e7"
Nov 01 08:54:07.335216 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:07.335109    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:08.841621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.841584    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"]
Nov 01 08:54:08.843137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.843111    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d8b8240b-c0c8-4d14-8421-5a84f1caea3d" containerName="revoke-keys"
Nov 01 08:54:08.843352 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.843312    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b8240b-c0c8-4d14-8421-5a84f1caea3d" containerName="revoke-keys"
Nov 01 08:54:08.843519 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.843505    4314 cpu_manager.go:414] "containerMap: removing container" podUID="642b038d-db97-4f6b-b976-191dd2a175ea" containerName="revoke-keys"
Nov 01 08:54:08.843791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.843774    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642b038d-db97-4f6b-b976-191dd2a175ea" containerName="revoke-keys"
Nov 01 08:54:08.844055 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.844037    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="642b038d-db97-4f6b-b976-191dd2a175ea" containerName="revoke-keys"
Nov 01 08:54:08.844174 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.844161    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b8240b-c0c8-4d14-8421-5a84f1caea3d" containerName="revoke-keys"
Nov 01 08:54:08.862821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.862796    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:08.893994 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:08.893959    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"]
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.127860    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.128051    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.128161    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.128258    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.135249    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf7l7\" (UniqueName: \"kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.135322    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.211673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.135413    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.341558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.331921    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.341558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.332140    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7l7\" (UniqueName: \"kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.341558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.332220    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.341558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.333156    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.358636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.345161    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.358636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.345558    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:54:09.360022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.359987    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.411659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.411609    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.488616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.460389    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.488616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.460797    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:09.530575 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.519400    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7l7\" (UniqueName: \"kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.548019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.547984    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef83188e-87pft\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.639291 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.639252    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.763752 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.763608    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.849782 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.849731    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:09.855422 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:09.854616    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:10.350671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.350626    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:10.351270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.351241    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:54:10.352067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.352046    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:54:10.352259 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.352243    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:54:10.397569 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.397513    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:10.426214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.425876    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:10.431195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.430894    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:54:10.436856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.436837    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:54:10.465555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.464612    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:10.465555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.464974    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:54:10.916583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.913756    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:10.991708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:10.969217    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:54:11.088707 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.088650    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:11.187172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.185228    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:11.236083 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.236058    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:54:11.236335 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.236304    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:54:11.243901 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.243879    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"]
Nov 01 08:54:11.244113 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.244090    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:54:11.244279 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.244266    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:54:11.244396 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.244385    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:54:11.248274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.248256    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:54:11.324593 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:54:11.324543    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:54:11.505448 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.505416    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"]
Nov 01 08:54:11.511368 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.511347    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.564795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.564742    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"]
Nov 01 08:54:11.616935 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.616904    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.617107 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.617087    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv65\" (UniqueName: \"kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.617195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.617178    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.718453 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.718417    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv65\" (UniqueName: \"kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.718602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.718527    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.718696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.718679    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.719107 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.719079    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.720898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.720874    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.777673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.777601    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv65\" (UniqueName: \"kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65\") pod \"maas-api-revoke-sub-9daa0279-2f5mc\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:11.824920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:11.824891    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:12.100763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.100678    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft" event={"ID":"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9","Type":"ContainerStarted","Data":"0b007855f5b6e12ce11bff60feb62ef2410a8e22a444bf6757db3cc50543b66f"}
Nov 01 08:54:12.101189 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.100764    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft" event={"ID":"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9","Type":"ContainerStarted","Data":"46e02538dfd6f819ca4fed8975b1b66ade757f0409b171b3de7f9cdea24945ac"}
Nov 01 08:54:12.259288 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.259229    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft" podStartSLOduration=4.259213756 podStartE2EDuration="4.259213756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:12.257273497 +0000 UTC m=+1325.451668053" watchObservedRunningTime="2026-11-01 08:54:12.259213756 +0000 UTC m=+1325.453608311"
Nov 01 08:54:12.309796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.309769    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"]
Nov 01 08:54:12.337409 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:12.337378    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb422f0d6_af8e_444d_acfc_cc1925db1fc2.slice/crio-485230cc850dd2eb553948cd166086b33de93ce849f00d4321e8f2d9f644056c WatchSource:0}: Error finding container 485230cc850dd2eb553948cd166086b33de93ce849f00d4321e8f2d9f644056c: Status 404 returned error can't find the container with id 485230cc850dd2eb553948cd166086b33de93ce849f00d4321e8f2d9f644056c
Nov 01 08:54:12.579412 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.579381    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"]
Nov 01 08:54:12.584983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.584960    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.637029 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.636968    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"]
Nov 01 08:54:12.732589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.732563    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.732697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.732668    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppc2\" (UniqueName: \"kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.732746 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.732703    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.834036 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.834007    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.834181 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.834066    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppc2\" (UniqueName: \"kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.834181 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.834090    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.834811 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.834793    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.836380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.836349    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.891790 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.891740    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppc2\" (UniqueName: \"kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2\") pod \"maas-api-revoke-sub-48b7302c-wnckc\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:12.898573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:12.898528    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:13.109526 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:13.109485    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc" event={"ID":"b422f0d6-af8e-444d-acfc-cc1925db1fc2","Type":"ContainerStarted","Data":"300e3f582a794132685b22ae0088bd4b08e77ef04d10341bbaf2e31a87929e64"}
Nov 01 08:54:13.109980 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:13.109584    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc" event={"ID":"b422f0d6-af8e-444d-acfc-cc1925db1fc2","Type":"ContainerStarted","Data":"485230cc850dd2eb553948cd166086b33de93ce849f00d4321e8f2d9f644056c"}
Nov 01 08:54:13.111245 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:13.111219    4314 generic.go:356] "Generic (PLEG): container finished" podID="d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" containerID="0b007855f5b6e12ce11bff60feb62ef2410a8e22a444bf6757db3cc50543b66f" exitCode=0
Nov 01 08:54:13.111353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:13.111297    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft" event={"ID":"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9","Type":"ContainerDied","Data":"0b007855f5b6e12ce11bff60feb62ef2410a8e22a444bf6757db3cc50543b66f"}
Nov 01 08:54:13.221262 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:13.221174    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc" podStartSLOduration=2.221160127 podStartE2EDuration="2.221160127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:13.218576446 +0000 UTC m=+1326.412971001" watchObservedRunningTime="2026-11-01 08:54:13.221160127 +0000 UTC m=+1326.415554673"
Nov 01 08:54:13.269861 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:13.269830    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb74d753_b538_4b8b_8075_8815d0674ba6.slice/crio-d6bb94d9370512cd92e20dae9cd4aeaf91a4743492e3cffd26739cbd101eaafc WatchSource:0}: Error finding container d6bb94d9370512cd92e20dae9cd4aeaf91a4743492e3cffd26739cbd101eaafc: Status 404 returned error can't find the container with id d6bb94d9370512cd92e20dae9cd4aeaf91a4743492e3cffd26739cbd101eaafc
Nov 01 08:54:13.269971 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:13.269890    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"]
Nov 01 08:54:14.127242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.127209    4314 generic.go:356] "Generic (PLEG): container finished" podID="b422f0d6-af8e-444d-acfc-cc1925db1fc2" containerID="300e3f582a794132685b22ae0088bd4b08e77ef04d10341bbaf2e31a87929e64" exitCode=0
Nov 01 08:54:14.127664 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.127300    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc" event={"ID":"b422f0d6-af8e-444d-acfc-cc1925db1fc2","Type":"ContainerDied","Data":"300e3f582a794132685b22ae0088bd4b08e77ef04d10341bbaf2e31a87929e64"}
Nov 01 08:54:14.129008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.128982    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc" event={"ID":"eb74d753-b538-4b8b-8075-8815d0674ba6","Type":"ContainerStarted","Data":"5255bf2e4900f3fce4021e82c1c16c330f8ebceb98e176c16f2e23527cd59904"}
Nov 01 08:54:14.129119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.129024    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc" event={"ID":"eb74d753-b538-4b8b-8075-8815d0674ba6","Type":"ContainerStarted","Data":"d6bb94d9370512cd92e20dae9cd4aeaf91a4743492e3cffd26739cbd101eaafc"}
Nov 01 08:54:14.346713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.346512    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc" podStartSLOduration=2.346498384 podStartE2EDuration="2.346498384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:14.342670934 +0000 UTC m=+1327.537065490" watchObservedRunningTime="2026-11-01 08:54:14.346498384 +0000 UTC m=+1327.540892938"
Nov 01 08:54:14.359085 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.359067    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:14.455690 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.455639    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7\" (UniqueName: \"kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7\") pod \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") "
Nov 01 08:54:14.455802 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.455705    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca\") pod \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") "
Nov 01 08:54:14.455870 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.455803    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca\") pod \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\" (UID: \"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9\") "
Nov 01 08:54:14.456097 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.456064    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca" pod "d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" (UID: "d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:14.456361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.456344    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:14.457674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.457644    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7" pod "d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" (UID: "d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9"). InnerVolumeSpecName "kube-api-access-nf7l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:14.457744 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.457714    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca" pod "d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" (UID: "d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:14.557757 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.557726    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf7l7\" (UniqueName: \"kubernetes.io/projected/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-kube-api-access-nf7l7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:14.557757 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:14.557753    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:15.137002 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.136966    4314 generic.go:356] "Generic (PLEG): container finished" podID="eb74d753-b538-4b8b-8075-8815d0674ba6" containerID="5255bf2e4900f3fce4021e82c1c16c330f8ebceb98e176c16f2e23527cd59904" exitCode=0
Nov 01 08:54:15.137449 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.137050    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc" event={"ID":"eb74d753-b538-4b8b-8075-8815d0674ba6","Type":"ContainerDied","Data":"5255bf2e4900f3fce4021e82c1c16c330f8ebceb98e176c16f2e23527cd59904"}
Nov 01 08:54:15.138484 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.138463    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"
Nov 01 08:54:15.138629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.138503    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft" event={"ID":"d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9","Type":"ContainerDied","Data":"46e02538dfd6f819ca4fed8975b1b66ade757f0409b171b3de7f9cdea24945ac"}
Nov 01 08:54:15.138629 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.138558    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e02538dfd6f819ca4fed8975b1b66ade757f0409b171b3de7f9cdea24945ac"
Nov 01 08:54:15.308986 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.308955    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:15.469687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.469597    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca\") pod \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") "
Nov 01 08:54:15.469687 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.469663    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65\" (UniqueName: \"kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65\") pod \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") "
Nov 01 08:54:15.469918 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.469783    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca\") pod \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\" (UID: \"b422f0d6-af8e-444d-acfc-cc1925db1fc2\") "
Nov 01 08:54:15.470126 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.470099    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca" pod "b422f0d6-af8e-444d-acfc-cc1925db1fc2" (UID: "b422f0d6-af8e-444d-acfc-cc1925db1fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:15.470314 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.470298    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:15.471743 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.471722    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65" pod "b422f0d6-af8e-444d-acfc-cc1925db1fc2" (UID: "b422f0d6-af8e-444d-acfc-cc1925db1fc2"). InnerVolumeSpecName "kube-api-access-7nv65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:15.471837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.471795    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca" pod "b422f0d6-af8e-444d-acfc-cc1925db1fc2" (UID: "b422f0d6-af8e-444d-acfc-cc1925db1fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:15.571512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.571483    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b422f0d6-af8e-444d-acfc-cc1925db1fc2-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:15.571512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.571511    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nv65\" (UniqueName: \"kubernetes.io/projected/b422f0d6-af8e-444d-acfc-cc1925db1fc2-kube-api-access-7nv65\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:15.997929 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:15.997890    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:16.147612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.147580    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"
Nov 01 08:54:16.149151 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.147644    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc" event={"ID":"b422f0d6-af8e-444d-acfc-cc1925db1fc2","Type":"ContainerDied","Data":"485230cc850dd2eb553948cd166086b33de93ce849f00d4321e8f2d9f644056c"}
Nov 01 08:54:16.149386 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.149205    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485230cc850dd2eb553948cd166086b33de93ce849f00d4321e8f2d9f644056c"
Nov 01 08:54:16.319768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.319738    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:16.483254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.483222    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca\") pod \"eb74d753-b538-4b8b-8075-8815d0674ba6\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") "
Nov 01 08:54:16.483410 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.483358    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca\") pod \"eb74d753-b538-4b8b-8075-8815d0674ba6\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") "
Nov 01 08:54:16.483410 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.483394    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2\" (UniqueName: \"kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2\") pod \"eb74d753-b538-4b8b-8075-8815d0674ba6\" (UID: \"eb74d753-b538-4b8b-8075-8815d0674ba6\") "
Nov 01 08:54:16.483617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.483594    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca" pod "eb74d753-b538-4b8b-8075-8815d0674ba6" (UID: "eb74d753-b538-4b8b-8075-8815d0674ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:16.484059 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.484032    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:16.485402 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.485376    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca" pod "eb74d753-b538-4b8b-8075-8815d0674ba6" (UID: "eb74d753-b538-4b8b-8075-8815d0674ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:16.485476 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.485441    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2" pod "eb74d753-b538-4b8b-8075-8815d0674ba6" (UID: "eb74d753-b538-4b8b-8075-8815d0674ba6"). InnerVolumeSpecName "kube-api-access-4ppc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:16.585295 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.585229    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb74d753-b538-4b8b-8075-8815d0674ba6-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:16.585295 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:16.585255    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ppc2\" (UniqueName: \"kubernetes.io/projected/eb74d753-b538-4b8b-8075-8815d0674ba6-kube-api-access-4ppc2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:17.156096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:17.156065    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc" event={"ID":"eb74d753-b538-4b8b-8075-8815d0674ba6","Type":"ContainerDied","Data":"d6bb94d9370512cd92e20dae9cd4aeaf91a4743492e3cffd26739cbd101eaafc"}
Nov 01 08:54:17.156096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:17.156089    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"
Nov 01 08:54:17.156591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:17.156112    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6bb94d9370512cd92e20dae9cd4aeaf91a4743492e3cffd26739cbd101eaafc"
Nov 01 08:54:20.775876 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:20.775840    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:21.965844 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:21.965797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:22.596926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:22.596880    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:22.891699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:22.891619    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:23.408030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:23.407989    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:23.550795 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:23.550755    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:23.707605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:23.707485    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:24.050596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:24.050556    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"]
Nov 01 08:54:24.096756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:24.096717    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37ca150-q528f"]
Nov 01 08:54:24.753542 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:24.753511    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:24.900674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:24.900635    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:24.929173 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:24.929140    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06cee309-adc9-4520-ae57-41462a339e27" path="/var/lib/kubelet/pods/06cee309-adc9-4520-ae57-41462a339e27/volumes"
Nov 01 08:54:25.059367 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:25.059291    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:25.732635 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:25.732596    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:27.170779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:27.170726    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:27.361899 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:27.361864    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:27.515112 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:27.515065    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:29.627822 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:29.627768    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:29.751710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:29.751669    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:29.855677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:29.855628    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:29.972804 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:29.972689    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:30.167181 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.165841    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.255557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.239308    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:30.255557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.239753    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.256718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.256654    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:30.460025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.459981    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:30.629197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.629063    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.632008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.630557    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:54:30.764065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.762151    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:30.884369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.884257    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.902558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.886155    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:54:30.953487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.934685    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Nov 01 08:54:30.978624 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:30.975616    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.000623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.000587    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:31.155654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.155375    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:54:31.207554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.168674    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.207554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.182667    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:54:31.269310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.246775    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.269310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.247603    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:54:31.313563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.270646    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.313563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.271389    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:54:31.313563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.299336    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.313563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.300014    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:54:31.629556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.629143    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.654554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.651693    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:54:31.708558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.707636    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 08:54:31.736557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.735789    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.789217    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"]
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790077    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790100    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790135    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b422f0d6-af8e-444d-acfc-cc1925db1fc2" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790144    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b422f0d6-af8e-444d-acfc-cc1925db1fc2" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790165    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eb74d753-b538-4b8b-8075-8815d0674ba6" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790173    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb74d753-b538-4b8b-8075-8815d0674ba6" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790335    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b422f0d6-af8e-444d-acfc-cc1925db1fc2" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790347    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb74d753-b538-4b8b-8075-8815d0674ba6" containerName="revoke-keys"
Nov 01 08:54:31.820557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.790364    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" containerName="revoke-keys"
Nov 01 08:54:31.852516 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.852477    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:31.897622 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.892301    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.917768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.917734    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"]
Nov 01 08:54:31.927116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.927083    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d31acd45-dockercfg-plj2z\""
Nov 01 08:54:31.963559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.931455    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d31acd45\""
Nov 01 08:54:31.979697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.979653    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.994567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.993394    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:31.994567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:31.993616    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89kx\" (UniqueName: \"kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:32.114560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.100506    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89kx\" (UniqueName: \"kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:32.147562 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.139319    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:32.232614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.194458    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:54:32.232614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.197634    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:32.296557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.260919    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.314557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.309739    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"]
Nov 01 08:54:32.446555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.310385    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:54:32.446555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.310476    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:32.446555 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.313224    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89kx\" (UniqueName: \"kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:32.491755 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.472742    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"]
Nov 01 08:54:32.492013 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.490183    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:32.603547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.603136    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.631770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.612205    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7xm\" (UniqueName: \"kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:32.631770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.612266    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:32.631770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.613066    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:32.713678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.679938    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.992263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.775756    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 08:54:32.992263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.828891    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7xm\" (UniqueName: \"kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:32.992263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.828965    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:32.992263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.830072    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:32.992263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:32.898008    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.048828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.016209    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.184547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.184498    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7xm\" (UniqueName: \"kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:33.206700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.195668    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.339081 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.339037    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.365156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.363979    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:54:33.365156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.364057    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:54:33.365156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.364107    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:54:33.365156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.364161    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:54:33.365156 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.364185    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"]
Nov 01 08:54:33.384882 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.378054    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:33.466234 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.465677    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"]
Nov 01 08:54:33.466487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.465762    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.480268 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.478969    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:54:33.499525 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.499463    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxl7\" (UniqueName: \"kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.515131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.515098    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.515514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.515490    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.539571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.539523    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:54:33.539873 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.539854    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:54:33.540121 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.540103    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:54:33.540514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.540353    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:54:33.617807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.617773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.620856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.620826    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxl7\" (UniqueName: \"kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.621558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.621521    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.622564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.622372    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.670624 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.664802    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:54:33.672640 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.671254    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:54:33.676445 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.676372    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:33.683611 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.683495    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxl7\" (UniqueName: \"kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.695066 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.693046    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:54:33.707706 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.707672    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"]
Nov 01 08:54:33.711384 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.711313    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bdb9f64-m9mwg\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:33.722111 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:33.722077    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c136930_d835_4535_b336_9ebae7bb8fe7.slice/crio-5492453208b5dca4c7180de364ee6d858f73d26ab237ea0bb2ea527562689c0d WatchSource:0}: Error finding container 5492453208b5dca4c7180de364ee6d858f73d26ab237ea0bb2ea527562689c0d: Status 404 returned error can't find the container with id 5492453208b5dca4c7180de364ee6d858f73d26ab237ea0bb2ea527562689c0d
Nov 01 08:54:33.978971 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:33.978943    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:34.065934 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:34.065904    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3e831d_b67a_48d8_b593_23274bcf3630.slice/crio-16a7d96378a1be6658a48fc6d618e98db69a9896f3664479f5ea49078bea0567 WatchSource:0}: Error finding container 16a7d96378a1be6658a48fc6d618e98db69a9896f3664479f5ea49078bea0567: Status 404 returned error can't find the container with id 16a7d96378a1be6658a48fc6d618e98db69a9896f3664479f5ea49078bea0567
Nov 01 08:54:34.068826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.068797    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"]
Nov 01 08:54:34.357131 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:34.357088    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dd4db2_d800_4158_940c_903b2c8f591e.slice/crio-161dd11126955ad3b4994a030c3a33d21a32e8ae97eb1f91a7e543220730d42b WatchSource:0}: Error finding container 161dd11126955ad3b4994a030c3a33d21a32e8ae97eb1f91a7e543220730d42b: Status 404 returned error can't find the container with id 161dd11126955ad3b4994a030c3a33d21a32e8ae97eb1f91a7e543220730d42b
Nov 01 08:54:34.359019 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.358978    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"]
Nov 01 08:54:34.370093 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.370067    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" event={"ID":"2c136930-d835-4535-b336-9ebae7bb8fe7","Type":"ContainerStarted","Data":"d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411"}
Nov 01 08:54:34.370202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.370123    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" event={"ID":"2c136930-d835-4535-b336-9ebae7bb8fe7","Type":"ContainerStarted","Data":"5492453208b5dca4c7180de364ee6d858f73d26ab237ea0bb2ea527562689c0d"}
Nov 01 08:54:34.370797 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.370780    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:34.372511 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.372491    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" event={"ID":"6b3e831d-b67a-48d8-b593-23274bcf3630","Type":"ContainerStarted","Data":"ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00"}
Nov 01 08:54:34.372596 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.372528    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" event={"ID":"6b3e831d-b67a-48d8-b593-23274bcf3630","Type":"ContainerStarted","Data":"16a7d96378a1be6658a48fc6d618e98db69a9896f3664479f5ea49078bea0567"}
Nov 01 08:54:34.373012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.372996    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:34.374786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.374765    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg" event={"ID":"44dd4db2-d800-4158-940c-903b2c8f591e","Type":"ContainerStarted","Data":"161dd11126955ad3b4994a030c3a33d21a32e8ae97eb1f91a7e543220730d42b"}
Nov 01 08:54:34.550609 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.550556    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" podStartSLOduration=3.550528421 podStartE2EDuration="3.550528421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:34.548781866 +0000 UTC m=+1347.743176420" watchObservedRunningTime="2026-11-01 08:54:34.550528421 +0000 UTC m=+1347.744922975"
Nov 01 08:54:34.658099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:34.658007    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" podStartSLOduration=3.657993635 podStartE2EDuration="3.657993635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:34.656737258 +0000 UTC m=+1347.851131812" watchObservedRunningTime="2026-11-01 08:54:34.657993635 +0000 UTC m=+1347.852388190"
Nov 01 08:54:35.383645 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:35.383611    4314 generic.go:356] "Generic (PLEG): container finished" podID="44dd4db2-d800-4158-940c-903b2c8f591e" containerID="b629e620b7f86626acf1b13566b29ab8b773c4d3dbcb0f3dd3d0544c9a99be03" exitCode=0
Nov 01 08:54:35.383995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:35.383702    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg" event={"ID":"44dd4db2-d800-4158-940c-903b2c8f591e","Type":"ContainerDied","Data":"b629e620b7f86626acf1b13566b29ab8b773c4d3dbcb0f3dd3d0544c9a99be03"}
Nov 01 08:54:36.545458 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.545432    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:36.665617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.665546    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca\") pod \"44dd4db2-d800-4158-940c-903b2c8f591e\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") "
Nov 01 08:54:36.665757 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.665618    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7\" (UniqueName: \"kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7\") pod \"44dd4db2-d800-4158-940c-903b2c8f591e\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") "
Nov 01 08:54:36.665918 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.665896    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca\") pod \"44dd4db2-d800-4158-940c-903b2c8f591e\" (UID: \"44dd4db2-d800-4158-940c-903b2c8f591e\") "
Nov 01 08:54:36.666315 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.666274    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca" pod "44dd4db2-d800-4158-940c-903b2c8f591e" (UID: "44dd4db2-d800-4158-940c-903b2c8f591e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:36.666513 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.666489    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:36.667779 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.667751    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca" pod "44dd4db2-d800-4158-940c-903b2c8f591e" (UID: "44dd4db2-d800-4158-940c-903b2c8f591e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:36.667923 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.667903    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7" pod "44dd4db2-d800-4158-940c-903b2c8f591e" (UID: "44dd4db2-d800-4158-940c-903b2c8f591e"). InnerVolumeSpecName "kube-api-access-vjxl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:36.767689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.767663    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxl7\" (UniqueName: \"kubernetes.io/projected/44dd4db2-d800-4158-940c-903b2c8f591e-kube-api-access-vjxl7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:36.767689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:36.767688    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44dd4db2-d800-4158-940c-903b2c8f591e-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:37.402564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:37.402509    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg" event={"ID":"44dd4db2-d800-4158-940c-903b2c8f591e","Type":"ContainerDied","Data":"161dd11126955ad3b4994a030c3a33d21a32e8ae97eb1f91a7e543220730d42b"}
Nov 01 08:54:37.402725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:37.402595    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161dd11126955ad3b4994a030c3a33d21a32e8ae97eb1f91a7e543220730d42b"
Nov 01 08:54:37.402725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:37.402663    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"
Nov 01 08:54:38.566580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:38.566549    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:40.164407 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.164370    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"]
Nov 01 08:54:40.165135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.165113    4314 cpu_manager.go:414] "containerMap: removing container" podUID="44dd4db2-d800-4158-940c-903b2c8f591e" containerName="revoke-keys"
Nov 01 08:54:40.165224 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.165136    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44dd4db2-d800-4158-940c-903b2c8f591e" containerName="revoke-keys"
Nov 01 08:54:40.165348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.165334    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="44dd4db2-d800-4158-940c-903b2c8f591e" containerName="revoke-keys"
Nov 01 08:54:40.169088 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.169067    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.223474 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.223441    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"]
Nov 01 08:54:40.263295 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.263270    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:54:40.263418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.263314    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:54:40.263418 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.263347    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:54:40.264168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.264150    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:54:40.309778 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.309755    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.309909 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.309788    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.310015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.309997    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qct\" (UniqueName: \"kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.411524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.411491    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.411661 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.411648    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qct\" (UniqueName: \"kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.411792 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.411777    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.412405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.412383    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.413836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.413818    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.447639 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.447586    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"]
Nov 01 08:54:40.447800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.447767    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy" containerID="cri-o://407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7" gracePeriod=30
Nov 01 08:54:40.469387 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.469363    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qct\" (UniqueName: \"kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct\") pod \"maas-api-revoke-sub-9b4bdcd1-l92d4\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.482168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.482147    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:40.859389 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:40.859357    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b7ad41_635e_497b_a9d5_ba1294ee0fff.slice/crio-b179efb002290bc2396a9858f376368862f8bc1a24771903e922cd7bda44472e WatchSource:0}: Error finding container b179efb002290bc2396a9858f376368862f8bc1a24771903e922cd7bda44472e: Status 404 returned error can't find the container with id b179efb002290bc2396a9858f376368862f8bc1a24771903e922cd7bda44472e
Nov 01 08:54:40.859880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:40.859849    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"]
Nov 01 08:54:41.322821 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:54:41.322784    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:54:41.418875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:41.418800    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:41.433379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:41.433345    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4" event={"ID":"c7b7ad41-635e-497b-a9d5-ba1294ee0fff","Type":"ContainerStarted","Data":"277d51acc216418e37ec5daa7994d8a83f4cba406c79ee55f499ea19615c3e12"}
Nov 01 08:54:41.433515 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:41.433406    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4" event={"ID":"c7b7ad41-635e-497b-a9d5-ba1294ee0fff","Type":"ContainerStarted","Data":"b179efb002290bc2396a9858f376368862f8bc1a24771903e922cd7bda44472e"}
Nov 01 08:54:41.542008 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:41.541956    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4" podStartSLOduration=1.5419425439999999 podStartE2EDuration="1.541942544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:41.54161567 +0000 UTC m=+1354.736010226" watchObservedRunningTime="2026-11-01 08:54:41.541942544 +0000 UTC m=+1354.736337099"
Nov 01 08:54:42.447728 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:42.447689    4314 generic.go:356] "Generic (PLEG): container finished" podID="c7b7ad41-635e-497b-a9d5-ba1294ee0fff" containerID="277d51acc216418e37ec5daa7994d8a83f4cba406c79ee55f499ea19615c3e12" exitCode=0
Nov 01 08:54:42.448252 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:42.447737    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4" event={"ID":"c7b7ad41-635e-497b-a9d5-ba1294ee0fff","Type":"ContainerDied","Data":"277d51acc216418e37ec5daa7994d8a83f4cba406c79ee55f499ea19615c3e12"}
Nov 01 08:54:43.323135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.323103    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:43.466108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.466076    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:43.641379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.641351    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:43.755279 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.755250    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca\") pod \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") "
Nov 01 08:54:43.755424 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.755310    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca\") pod \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") "
Nov 01 08:54:43.755424 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.755334    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct\" (UniqueName: \"kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct\") pod \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\" (UID: \"c7b7ad41-635e-497b-a9d5-ba1294ee0fff\") "
Nov 01 08:54:43.755691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.755665    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca" pod "c7b7ad41-635e-497b-a9d5-ba1294ee0fff" (UID: "c7b7ad41-635e-497b-a9d5-ba1294ee0fff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:43.756011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.755985    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:43.757637 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.757610    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca" pod "c7b7ad41-635e-497b-a9d5-ba1294ee0fff" (UID: "c7b7ad41-635e-497b-a9d5-ba1294ee0fff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:43.757763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.757745    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct" pod "c7b7ad41-635e-497b-a9d5-ba1294ee0fff" (UID: "c7b7ad41-635e-497b-a9d5-ba1294ee0fff"). InnerVolumeSpecName "kube-api-access-s6qct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:43.857000 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.856970    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:43.857000 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:43.856999    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6qct\" (UniqueName: \"kubernetes.io/projected/c7b7ad41-635e-497b-a9d5-ba1294ee0fff-kube-api-access-s6qct\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:44.245310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.245279    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"]
Nov 01 08:54:44.245908 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.245892    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c7b7ad41-635e-497b-a9d5-ba1294ee0fff" containerName="revoke-keys"
Nov 01 08:54:44.245949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.245910    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b7ad41-635e-497b-a9d5-ba1294ee0fff" containerName="revoke-keys"
Nov 01 08:54:44.246014 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.246004    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b7ad41-635e-497b-a9d5-ba1294ee0fff" containerName="revoke-keys"
Nov 01 08:54:44.249689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.249673    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.298768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.298740    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 08:54:44.298871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.298775    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 08:54:44.298871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.298819    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 08:54:44.344559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.344505    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z5ppc\""
Nov 01 08:54:44.350604 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.350581    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"]
Nov 01 08:54:44.364983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.364958    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.365147 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.365125    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.365225 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.365203    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.365291 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.365272    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzft\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.365405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.365378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.365510 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.365493    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.365598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.365581    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466408 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466383    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466433    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466562    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466678    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466718    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466776    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzft\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.466895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466820    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.467217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466951    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4" event={"ID":"c7b7ad41-635e-497b-a9d5-ba1294ee0fff","Type":"ContainerDied","Data":"b179efb002290bc2396a9858f376368862f8bc1a24771903e922cd7bda44472e"}
Nov 01 08:54:44.467217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.466996    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b179efb002290bc2396a9858f376368862f8bc1a24771903e922cd7bda44472e"
Nov 01 08:54:44.467217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.467053    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"
Nov 01 08:54:44.467370 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.467348    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.469074 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.469041    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.469225 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.469203    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.469421 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.469403    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.469694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.469673    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.523595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.523523    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.523722 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.523708    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzft\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5vst\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.562908 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.562890    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:44.982680 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:44.982652    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fa35a9_077b_43d2_bd27_ee185f40f9cd.slice/crio-88effa0d450fbfe711799dd60a48faa6a9f37cf4df2b9d6653f2981191d48d39 WatchSource:0}: Error finding container 88effa0d450fbfe711799dd60a48faa6a9f37cf4df2b9d6653f2981191d48d39: Status 404 returned error can't find the container with id 88effa0d450fbfe711799dd60a48faa6a9f37cf4df2b9d6653f2981191d48d39
Nov 01 08:54:44.982972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.982950    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"]
Nov 01 08:54:44.985065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:44.985045    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:54:45.478769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.478740    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" event={"ID":"a8fa35a9-077b-43d2-bd27-ee185f40f9cd","Type":"ContainerStarted","Data":"88effa0d450fbfe711799dd60a48faa6a9f37cf4df2b9d6653f2981191d48d39"}
Nov 01 08:54:45.720373 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.720313    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883654    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883711    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883741    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883778    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883845    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883893    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883917    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.883982    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.884053    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.884137    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97\") pod \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\" (UID: \"468b79bd-a23b-4047-b56e-a1e5c515bcc0\") "
Nov 01 08:54:45.884856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.884493    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:54:45.884907 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.884873    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.884907 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.884897    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.885313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.885284    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:45.885711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.885686    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:54:45.885829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.885785    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:54:45.886920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.886896    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:45.887099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.887065    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "kube-api-access-ppx97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:45.887325 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.887307    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:54:45.887880 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.887855    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo" pod "468b79bd-a23b-4047-b56e-a1e5c515bcc0" (UID: "468b79bd-a23b-4047-b56e-a1e5c515bcc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 08:54:45.986143 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986119    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.986143 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986145    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.986286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986155    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.986286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986165    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.986286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986174    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppx97\" (UniqueName: \"kubernetes.io/projected/468b79bd-a23b-4047-b56e-a1e5c515bcc0-kube-api-access-ppx97\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.986286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986185    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:45.986286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:45.986193    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468b79bd-a23b-4047-b56e-a1e5c515bcc0-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:46.396308 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.396230    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:54:46.396308 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.396307    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:54:46.487965 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.487933    4314 generic.go:356] "Generic (PLEG): container finished" podID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerID="407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7" exitCode=0
Nov 01 08:54:46.488391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.487982    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"
Nov 01 08:54:46.488391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.488011    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" event={"ID":"468b79bd-a23b-4047-b56e-a1e5c515bcc0","Type":"ContainerDied","Data":"407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7"}
Nov 01 08:54:46.488391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.488063    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv" event={"ID":"468b79bd-a23b-4047-b56e-a1e5c515bcc0","Type":"ContainerDied","Data":"8038e0196da7f96a0ff161a4b1826b9a7a9c78089bcde21b9d6a025498c08984"}
Nov 01 08:54:46.488391 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.488098    4314 scope.go:122] "RemoveContainer" containerID="407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7"
Nov 01 08:54:46.502167 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.502146    4314 scope.go:122] "RemoveContainer" containerID="407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7"
Nov 01 08:54:46.502543 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:54:46.502508    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7\": container with ID starting with 407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7 not found: ID does not exist" containerID="407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7"
Nov 01 08:54:46.502643 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.502555    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7"} err="failed to get container status \"407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7\": rpc error: code = NotFound desc = could not find container \"407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7\": container with ID starting with 407aeb7c94c3bf4902aef48b2a753d7f7eca800beda380bd801c3f2d297e64b7 not found: ID does not exist"
Nov 01 08:54:46.725865 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.725788    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"]
Nov 01 08:54:46.772069 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.772040    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-6vwdv"]
Nov 01 08:54:46.930760 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:46.930726    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" path="/var/lib/kubelet/pods/468b79bd-a23b-4047-b56e-a1e5c515bcc0/volumes"
Nov 01 08:54:48.193187 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:48.193137    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:49.051186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:49.051148    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:54:49.051332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:49.051231    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:54:49.389861 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:49.389775    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:49.514557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:49.514496    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" event={"ID":"a8fa35a9-077b-43d2-bd27-ee185f40f9cd","Type":"ContainerStarted","Data":"7b82e52b807787d0fb05751d51393da59310513e5b7a83b3cb8e2cd983e87171"}
Nov 01 08:54:49.514734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:49.514613    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:49.721966 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:49.721877    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" podStartSLOduration=1.656965122 podStartE2EDuration="5.721863389s" totalImagesPullingTime="4.064898267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:49.67307033 +0000 UTC m=+1362.867464884" watchObservedRunningTime="2026-11-01 08:54:49.721863389 +0000 UTC m=+1362.916257943"
Nov 01 08:54:50.425786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.425754    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"]
Nov 01 08:54:50.426373 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.426358    4314 cpu_manager.go:414] "containerMap: removing container" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy"
Nov 01 08:54:50.426416 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.426374    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy"
Nov 01 08:54:50.426564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.426476    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="468b79bd-a23b-4047-b56e-a1e5c515bcc0" containerName="istio-proxy"
Nov 01 08:54:50.430251 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.430236    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.447498 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.447471    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zc2\" (UniqueName: \"kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.447623 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.447596    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.447701 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.447686    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.484278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.484253    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"]
Nov 01 08:54:50.527574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.525392    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:54:50.527574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.525443    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:54:50.527574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.525443    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:54:50.527574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.525508    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:54:50.528677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.528648    4314 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-q5vst container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 08:54:50.528813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.528709    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" podUID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:54:50.548963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.548931    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.549219 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.549200    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zc2\" (UniqueName: \"kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.549334 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.549315    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.549898 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.549877    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.551349 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.551328    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.606407 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.606375    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zc2\" (UniqueName: \"kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2\") pod \"maas-api-revoke-sub-da49e37e-p5f8r\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.744102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.744073    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:50.918380 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.918348    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"]
Nov 01 08:54:50.923791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.923770    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:50.955379 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.955356    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:50.955491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.955401    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:50.955491 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.955474    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cxc\" (UniqueName: \"kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:50.977544 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:50.977509    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"]
Nov 01 08:54:51.056639 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.056557    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.056639 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.056615    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.056856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.056658    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cxc\" (UniqueName: \"kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.057398 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.057361    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.058961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.058936    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.113648 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.113623    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cxc\" (UniqueName: \"kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc\") pod \"maas-api-revoke-sub-f06533b1-7tm97\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.140039 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:51.140011    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d61ece_5dbf_4161_958f_6ac825bf40df.slice/crio-799be7a62f6351aaa9869b55ddfd7ea6dbbf5488b621a67f442c9d8d767a088d WatchSource:0}: Error finding container 799be7a62f6351aaa9869b55ddfd7ea6dbbf5488b621a67f442c9d8d767a088d: Status 404 returned error can't find the container with id 799be7a62f6351aaa9869b55ddfd7ea6dbbf5488b621a67f442c9d8d767a088d
Nov 01 08:54:51.140209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.140189    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"]
Nov 01 08:54:51.239134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.239106    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:51.535399 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.535368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r" event={"ID":"27d61ece-5dbf-4161-958f-6ac825bf40df","Type":"ContainerStarted","Data":"8dcc95616f011c0aa82195352ce888e2b821017c231088ec0d54a5b2b8654a41"}
Nov 01 08:54:51.535819 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.535420    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r" event={"ID":"27d61ece-5dbf-4161-958f-6ac825bf40df","Type":"ContainerStarted","Data":"799be7a62f6351aaa9869b55ddfd7ea6dbbf5488b621a67f442c9d8d767a088d"}
Nov 01 08:54:51.611847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.611819    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"]
Nov 01 08:54:51.611980 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:54:51.611911    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182b8531_8a90_4d44_ab97_3fa8239d300b.slice/crio-d21cef65fae8bea050728e5da485409bec1cfbc26d71e9aaf8ceb9ba23d13c71 WatchSource:0}: Error finding container d21cef65fae8bea050728e5da485409bec1cfbc26d71e9aaf8ceb9ba23d13c71: Status 404 returned error can't find the container with id d21cef65fae8bea050728e5da485409bec1cfbc26d71e9aaf8ceb9ba23d13c71
Nov 01 08:54:51.643973 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:51.643918    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r" podStartSLOduration=1.643901318 podStartE2EDuration="1.643901318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:51.642933691 +0000 UTC m=+1364.837328246" watchObservedRunningTime="2026-11-01 08:54:51.643901318 +0000 UTC m=+1364.838295876"
Nov 01 08:54:52.559769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:52.559041    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97" event={"ID":"182b8531-8a90-4d44-ab97-3fa8239d300b","Type":"ContainerStarted","Data":"c5d92945c877b183a10b10223813d6f884ec4df12ec81e204d27f196ee2d8552"}
Nov 01 08:54:52.559769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:52.559143    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97" event={"ID":"182b8531-8a90-4d44-ab97-3fa8239d300b","Type":"ContainerStarted","Data":"d21cef65fae8bea050728e5da485409bec1cfbc26d71e9aaf8ceb9ba23d13c71"}
Nov 01 08:54:52.617565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:52.610509    4314 generic.go:356] "Generic (PLEG): container finished" podID="27d61ece-5dbf-4161-958f-6ac825bf40df" containerID="8dcc95616f011c0aa82195352ce888e2b821017c231088ec0d54a5b2b8654a41" exitCode=0
Nov 01 08:54:52.617565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:52.610676    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r" event={"ID":"27d61ece-5dbf-4161-958f-6ac825bf40df","Type":"ContainerDied","Data":"8dcc95616f011c0aa82195352ce888e2b821017c231088ec0d54a5b2b8654a41"}
Nov 01 08:54:52.827272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:52.827144    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97" podStartSLOduration=2.8271225700000002 podStartE2EDuration="2.82712257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:52.709844755 +0000 UTC m=+1365.904239311" watchObservedRunningTime="2026-11-01 08:54:52.82712257 +0000 UTC m=+1366.021517154"
Nov 01 08:54:53.356576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:53.347776    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:53.547065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:53.545792    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 08:54:53.659128 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:53.659035    4314 generic.go:356] "Generic (PLEG): container finished" podID="182b8531-8a90-4d44-ab97-3fa8239d300b" containerID="c5d92945c877b183a10b10223813d6f884ec4df12ec81e204d27f196ee2d8552" exitCode=0
Nov 01 08:54:53.660217 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:53.660175    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97" event={"ID":"182b8531-8a90-4d44-ab97-3fa8239d300b","Type":"ContainerDied","Data":"c5d92945c877b183a10b10223813d6f884ec4df12ec81e204d27f196ee2d8552"}
Nov 01 08:54:54.206559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.205628    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.298556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.297624    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.317557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.314612    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.347559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.346627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.347559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.347097    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:54:54.681688 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.681639    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:54:54.682195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.682126    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:54:54.682195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.682174    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.732671 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.732624    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.897341 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:54.897258    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.209710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.209620    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.211218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.211192    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:54:55.287559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.284729    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:55.287559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.285991    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2\" (UniqueName: \"kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2\") pod \"27d61ece-5dbf-4161-958f-6ac825bf40df\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") "
Nov 01 08:54:55.287559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.286049    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca\") pod \"27d61ece-5dbf-4161-958f-6ac825bf40df\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") "
Nov 01 08:54:55.287559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.286160    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca\") pod \"27d61ece-5dbf-4161-958f-6ac825bf40df\" (UID: \"27d61ece-5dbf-4161-958f-6ac825bf40df\") "
Nov 01 08:54:55.294793 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.294755    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca" pod "27d61ece-5dbf-4161-958f-6ac825bf40df" (UID: "27d61ece-5dbf-4161-958f-6ac825bf40df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:55.302883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.300621    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.302883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.301232    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:54:55.324769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.315749    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.324769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.322877    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca" pod "27d61ece-5dbf-4161-958f-6ac825bf40df" (UID: "27d61ece-5dbf-4161-958f-6ac825bf40df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:55.324769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.322991    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:54:55.324769 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.324325    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2" pod "27d61ece-5dbf-4161-958f-6ac825bf40df" (UID: "27d61ece-5dbf-4161-958f-6ac825bf40df"). InnerVolumeSpecName "kube-api-access-66zc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:55.363465 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.363429    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.364849 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.364803    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:55.371664 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.371636    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:54:55.371793 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.371708    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:54:55.399938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.390018    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66zc2\" (UniqueName: \"kubernetes.io/projected/27d61ece-5dbf-4161-958f-6ac825bf40df-kube-api-access-66zc2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:55.399938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.390053    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:55.399938 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.390070    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d61ece-5dbf-4161-958f-6ac825bf40df-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:55.715837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.713381    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:54:55.739053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.739011    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.739480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.739302    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:54:55.740191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.740162    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:54:55.947133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.945703    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 08:54:55.947133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.946286    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:54:55.974646 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.974559    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r" event={"ID":"27d61ece-5dbf-4161-958f-6ac825bf40df","Type":"ContainerDied","Data":"799be7a62f6351aaa9869b55ddfd7ea6dbbf5488b621a67f442c9d8d767a088d"}
Nov 01 08:54:55.974883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.974864    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="799be7a62f6351aaa9869b55ddfd7ea6dbbf5488b621a67f442c9d8d767a088d"
Nov 01 08:54:55.975046 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:55.975032    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"
Nov 01 08:54:56.210661 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.210618    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.569575 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.566737    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.642566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.612489    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:56.642566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.613185    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.670556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.670494    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.727022 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.726973    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.814770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.798721    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.946734 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:56.946687    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.084556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.035800    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:57.543559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.541332    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.568559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.544712    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.584576 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.576863    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.592583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.590546    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc\" (UniqueName: \"kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc\") pod \"182b8531-8a90-4d44-ab97-3fa8239d300b\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") "
Nov 01 08:54:57.595102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.595074    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca\") pod \"182b8531-8a90-4d44-ab97-3fa8239d300b\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") "
Nov 01 08:54:57.595337 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.595319    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca\") pod \"182b8531-8a90-4d44-ab97-3fa8239d300b\" (UID: \"182b8531-8a90-4d44-ab97-3fa8239d300b\") "
Nov 01 08:54:57.614713 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.614622    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca" pod "182b8531-8a90-4d44-ab97-3fa8239d300b" (UID: "182b8531-8a90-4d44-ab97-3fa8239d300b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:54:57.654712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.654670    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.681934 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.681891    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.692509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.692471    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.780142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.780094    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.068142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.845621    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.068142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.845741    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca" pod "182b8531-8a90-4d44-ab97-3fa8239d300b" (UID: "182b8531-8a90-4d44-ab97-3fa8239d300b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:54:58.068142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:57.922177    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:58.068142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.029637    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.131028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.130995    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182b8531-8a90-4d44-ab97-3fa8239d300b-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:58.176958 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.176919    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc" pod "182b8531-8a90-4d44-ab97-3fa8239d300b" (UID: "182b8531-8a90-4d44-ab97-3fa8239d300b"). InnerVolumeSpecName "kube-api-access-v5cxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:54:58.232573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.232521    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5cxc\" (UniqueName: \"kubernetes.io/projected/182b8531-8a90-4d44-ab97-3fa8239d300b-kube-api-access-v5cxc\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:54:58.554574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.554449    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.590710 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.590662    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.664682 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.664634    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.694092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.694046    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.702442 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.701632    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.780967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.780933    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"
Nov 01 08:54:58.849629 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:54:58.782707    4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.849s"
Nov 01 08:54:58.856714 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.856627    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97" event={"ID":"182b8531-8a90-4d44-ab97-3fa8239d300b","Type":"ContainerDied","Data":"d21cef65fae8bea050728e5da485409bec1cfbc26d71e9aaf8ceb9ba23d13c71"}
Nov 01 08:54:58.856920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.856905    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d21cef65fae8bea050728e5da485409bec1cfbc26d71e9aaf8ceb9ba23d13c71"
Nov 01 08:54:58.857098 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.846723    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 01 08:54:58.883896 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:58.883863    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:59.642699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.642658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:59.725021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.724988    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 08:54:59.785950 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.785906    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 08:54:59.787677 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.787637    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 08:54:59.788219 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.788188    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:59.789039 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.788838    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 08:54:59.790817 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.790062    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 08:54:59.798568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.790090    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Nov 01 08:54:59.798568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.790113    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 08:54:59.798568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.790431    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:59.800336 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.800172    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:54:59.832060 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.832010    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:59.845956 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.845929    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 08:54:59.848221 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:54:59.848199    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 08:55:00.204203 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:00.204163    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 08:55:00.253566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:00.253514    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 08:55:00.446241 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:00.446177    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 08:55:00.448583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:00.446391    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:00.491565 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:00.490944    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 08:55:00.601072 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:00.599928    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 08:55:03.297235 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:03.297203    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:03.447179 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:03.447149    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:03.652111 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:03.652032    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:04.029036 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.028998    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:04.631910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.631876    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:04.739705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.739672    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"]
Nov 01 08:55:04.740286 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.740271    4314 cpu_manager.go:414] "containerMap: removing container" podUID="182b8531-8a90-4d44-ab97-3fa8239d300b" containerName="revoke-keys"
Nov 01 08:55:04.740345 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.740287    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182b8531-8a90-4d44-ab97-3fa8239d300b" containerName="revoke-keys"
Nov 01 08:55:04.740345 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.740301    4314 cpu_manager.go:414] "containerMap: removing container" podUID="27d61ece-5dbf-4161-958f-6ac825bf40df" containerName="revoke-keys"
Nov 01 08:55:04.740345 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.740307    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d61ece-5dbf-4161-958f-6ac825bf40df" containerName="revoke-keys"
Nov 01 08:55:04.740438 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.740410    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="182b8531-8a90-4d44-ab97-3fa8239d300b" containerName="revoke-keys"
Nov 01 08:55:04.740438 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.740419    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d61ece-5dbf-4161-958f-6ac825bf40df" containerName="revoke-keys"
Nov 01 08:55:04.744204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.744187    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.781282 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.781238    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.781470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.781370    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44l7c\" (UniqueName: \"kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.781621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.781598    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.783209 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.783173    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:04.815974 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.815937    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"]
Nov 01 08:55:04.865564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.860907    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:55:04.865564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.861025    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:55:04.865564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.861745    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:55:04.865564 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.861840    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:55:04.891454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.891355    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.891962 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.891934    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.892119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.892104    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7c\" (UniqueName: \"kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.893273 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.893253    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.901177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.901136    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:04.934941 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.934912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:04.949405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:04.949381    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44l7c\" (UniqueName: \"kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c\") pod \"maas-api-revoke-sub-ef2e41c0-vdstr\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:05.057983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:05.057945    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:05.442793 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:55:05.442761    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf117cc_8ab5_47f7_ab22_1a76da47a1bc.slice/crio-0ee86b1e28aff3e07833f398d57352f806ef2493ff2d3db900a3fc6692f7fd28 WatchSource:0}: Error finding container 0ee86b1e28aff3e07833f398d57352f806ef2493ff2d3db900a3fc6692f7fd28: Status 404 returned error can't find the container with id 0ee86b1e28aff3e07833f398d57352f806ef2493ff2d3db900a3fc6692f7fd28
Nov 01 08:55:05.443302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:05.443277    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"]
Nov 01 08:55:05.648025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:05.647991    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr" event={"ID":"baf117cc-8ab5-47f7-ab22-1a76da47a1bc","Type":"ContainerStarted","Data":"0ee86b1e28aff3e07833f398d57352f806ef2493ff2d3db900a3fc6692f7fd28"}
Nov 01 08:55:06.657472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:06.657381    4314 generic.go:356] "Generic (PLEG): container finished" podID="baf117cc-8ab5-47f7-ab22-1a76da47a1bc" containerID="b159f6a4b65c988ab4242d24e54ce7c3010ba35944b18ad777748b6d7ad47a18" exitCode=0
Nov 01 08:55:06.657472 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:06.657454    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr" event={"ID":"baf117cc-8ab5-47f7-ab22-1a76da47a1bc","Type":"ContainerDied","Data":"b159f6a4b65c988ab4242d24e54ce7c3010ba35944b18ad777748b6d7ad47a18"}
Nov 01 08:55:07.298307 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.298265    4314 scope.go:122] "RemoveContainer" containerID="6f1d2cb837231ea3f630d8cfe16483ce77ccf5b82578c41ea29c5a97a4c340b4"
Nov 01 08:55:07.308358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.308338    4314 scope.go:122] "RemoveContainer" containerID="f5665af640346f5ad879b1fae0d268ffdee07550ff273d87f6f47c185b531e84"
Nov 01 08:55:07.318115 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.318091    4314 scope.go:122] "RemoveContainer" containerID="c89d0e38a81dac14f215c813512edd07ff354e7008f131d3df9506d226f9f30d"
Nov 01 08:55:07.326895 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.326876    4314 scope.go:122] "RemoveContainer" containerID="071d1ae600bce92c4cff5a77e21ba860330f4c5eac38a96000571666cb215ff8"
Nov 01 08:55:07.335090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.335066    4314 scope.go:122] "RemoveContainer" containerID="c012abea063f50969fb3737bb870efd21731ede76ffa0cec4e3706128980d298"
Nov 01 08:55:07.343328 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.343312    4314 scope.go:122] "RemoveContainer" containerID="430ef1acad3317c43fc35973256a60ebeeb54af7cc3db79f3e0f9a13203d45b5"
Nov 01 08:55:07.600160 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.600057    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:07.834678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.834652    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:07.936114 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.936045    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca\") pod \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") "
Nov 01 08:55:07.936258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.936244    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c\" (UniqueName: \"kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c\") pod \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") "
Nov 01 08:55:07.936305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.936271    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca\") pod \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\" (UID: \"baf117cc-8ab5-47f7-ab22-1a76da47a1bc\") "
Nov 01 08:55:07.936695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.936668    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca" pod "baf117cc-8ab5-47f7-ab22-1a76da47a1bc" (UID: "baf117cc-8ab5-47f7-ab22-1a76da47a1bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:55:07.938050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.938029    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca" pod "baf117cc-8ab5-47f7-ab22-1a76da47a1bc" (UID: "baf117cc-8ab5-47f7-ab22-1a76da47a1bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:55:07.938223 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:07.938202    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c" pod "baf117cc-8ab5-47f7-ab22-1a76da47a1bc" (UID: "baf117cc-8ab5-47f7-ab22-1a76da47a1bc"). InnerVolumeSpecName "kube-api-access-44l7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:55:08.037664 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.037634    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44l7c\" (UniqueName: \"kubernetes.io/projected/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-kube-api-access-44l7c\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:08.037858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.037835    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:08.037858 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.037858    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf117cc-8ab5-47f7-ab22-1a76da47a1bc-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:08.675028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.674995    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"
Nov 01 08:55:08.675198 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.675027    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr" event={"ID":"baf117cc-8ab5-47f7-ab22-1a76da47a1bc","Type":"ContainerDied","Data":"0ee86b1e28aff3e07833f398d57352f806ef2493ff2d3db900a3fc6692f7fd28"}
Nov 01 08:55:08.675198 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.675083    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee86b1e28aff3e07833f398d57352f806ef2493ff2d3db900a3fc6692f7fd28"
Nov 01 08:55:08.976462 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:08.976375    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:09.163361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:09.163322    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:09.307101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:09.307074    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:09.821093 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:09.821060    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:11.327393 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:55:11.327350    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:55:12.155721 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:12.155686    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:12.320076 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:12.320041    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:14.892072 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:14.892037    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:17.595420 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:17.595381    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:17.740305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:17.740267    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:21.239905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:21.239818    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:24.476054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:24.476009    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:29.440065 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:29.440025    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:34.380673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:34.380636    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:39.393353 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:39.393318    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:41.323278 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:55:41.323242    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:55:41.398493 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.398462    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"]
Nov 01 08:55:41.399073 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.399059    4314 cpu_manager.go:414] "containerMap: removing container" podUID="baf117cc-8ab5-47f7-ab22-1a76da47a1bc" containerName="revoke-keys"
Nov 01 08:55:41.399116 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.399075    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf117cc-8ab5-47f7-ab22-1a76da47a1bc" containerName="revoke-keys"
Nov 01 08:55:41.399186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.399177    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf117cc-8ab5-47f7-ab22-1a76da47a1bc" containerName="revoke-keys"
Nov 01 08:55:41.402978 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.402962    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.410494 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.410079    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.410494 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.410122    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnxq\" (UniqueName: \"kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.410494 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.410346    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.451649 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.451626    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:55:41.456090 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.456063    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"]
Nov 01 08:55:41.496461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.496431    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:55:41.497573 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.497528    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:55:41.511654 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.511630    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.511834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.511670    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnxq\" (UniqueName: \"kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.511834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.511818    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.512254 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.512225    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.514096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.514079    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.552778 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.552753    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:55:41.568783 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.568760    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnxq\" (UniqueName: \"kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq\") pod \"maas-api-revoke-sub-3db8a280-hhmls\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:41.716495 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:41.716419    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:42.084325 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:55:42.084284    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9efc96_3391_462c_8dae_96263874ae42.slice/crio-b46a8ea64c9cedfb3061fba79ba84d163754497b0dee21dcc13010c109a19857 WatchSource:0}: Error finding container b46a8ea64c9cedfb3061fba79ba84d163754497b0dee21dcc13010c109a19857: Status 404 returned error can't find the container with id b46a8ea64c9cedfb3061fba79ba84d163754497b0dee21dcc13010c109a19857
Nov 01 08:55:42.085047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:42.085017    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"]
Nov 01 08:55:43.000979 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:43.000947    4314 generic.go:356] "Generic (PLEG): container finished" podID="eb9efc96-3391-462c-8dae-96263874ae42" containerID="0b3713a554439603cd745b9dcd440cbc9722f389b0bb78bd8025351bff42a8e8" exitCode=0
Nov 01 08:55:43.001356 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:43.001049    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls" event={"ID":"eb9efc96-3391-462c-8dae-96263874ae42","Type":"ContainerDied","Data":"0b3713a554439603cd745b9dcd440cbc9722f389b0bb78bd8025351bff42a8e8"}
Nov 01 08:55:43.001356 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:43.001100    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls" event={"ID":"eb9efc96-3391-462c-8dae-96263874ae42","Type":"ContainerStarted","Data":"b46a8ea64c9cedfb3061fba79ba84d163754497b0dee21dcc13010c109a19857"}
Nov 01 08:55:43.450294 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:43.450262    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:44.166708 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.166683    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:44.242232 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.242206    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca\") pod \"eb9efc96-3391-462c-8dae-96263874ae42\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") "
Nov 01 08:55:44.242381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.242238    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca\") pod \"eb9efc96-3391-462c-8dae-96263874ae42\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") "
Nov 01 08:55:44.242381 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.242274    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq\" (UniqueName: \"kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq\") pod \"eb9efc96-3391-462c-8dae-96263874ae42\" (UID: \"eb9efc96-3391-462c-8dae-96263874ae42\") "
Nov 01 08:55:44.242656 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.242632    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca" pod "eb9efc96-3391-462c-8dae-96263874ae42" (UID: "eb9efc96-3391-462c-8dae-96263874ae42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:55:44.242959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.242940    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:44.244435 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.244409    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca" pod "eb9efc96-3391-462c-8dae-96263874ae42" (UID: "eb9efc96-3391-462c-8dae-96263874ae42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:55:44.244572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.244479    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq" pod "eb9efc96-3391-462c-8dae-96263874ae42" (UID: "eb9efc96-3391-462c-8dae-96263874ae42"). InnerVolumeSpecName "kube-api-access-whnxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:55:44.343996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.343922    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9efc96-3391-462c-8dae-96263874ae42-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:44.343996 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.343947    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnxq\" (UniqueName: \"kubernetes.io/projected/eb9efc96-3391-462c-8dae-96263874ae42-kube-api-access-whnxq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:44.429919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.429881    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:44.599164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.599080    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:44.941251 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:44.941221    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:45.017172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:45.017145    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"
Nov 01 08:55:45.017172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:45.017151    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls" event={"ID":"eb9efc96-3391-462c-8dae-96263874ae42","Type":"ContainerDied","Data":"b46a8ea64c9cedfb3061fba79ba84d163754497b0dee21dcc13010c109a19857"}
Nov 01 08:55:45.017367 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:45.017192    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46a8ea64c9cedfb3061fba79ba84d163754497b0dee21dcc13010c109a19857"
Nov 01 08:55:49.381263 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:49.381226    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:53.535589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:53.535508    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:54.162553 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:54.160029    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:54.313332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:54.313295    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:54.466191 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:54.466113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:58.585470 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:58.585365    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:58.792031 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:58.791992    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:58.943719 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:58.943618    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:55:59.314727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:55:59.314680    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:01.390405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:01.390369    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:01.779527 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:01.779489    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:01.981844 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:01.981810    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:02.946383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:02.946349    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:03.097225 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:03.097190    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:04.686146 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:04.686102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:07.400260 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:07.400228    4314 scope.go:122] "RemoveContainer" containerID="4d7cbc4a4c9cebab4394295192cb29ff3bf631d6c7188bc309357c9e923eff7c"
Nov 01 08:56:07.895048 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:07.895016    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:09.687166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:09.687134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:11.328559 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:56:11.328509    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:56:14.195638 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.195603    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"]
Nov 01 08:56:14.196233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.196216    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eb9efc96-3391-462c-8dae-96263874ae42" containerName="revoke-keys"
Nov 01 08:56:14.196278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.196234    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9efc96-3391-462c-8dae-96263874ae42" containerName="revoke-keys"
Nov 01 08:56:14.196351 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.196342    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9efc96-3391-462c-8dae-96263874ae42" containerName="revoke-keys"
Nov 01 08:56:14.199920 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.199906    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.253854 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.253823    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"]
Nov 01 08:56:14.293867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.293831    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:56:14.293867 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.293837    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:56:14.294439 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.294419    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:56:14.294497 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.294441    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:56:14.323724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.323692    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:14.344456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.344425    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.344621 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.344466    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67nxm\" (UniqueName: \"kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.344707 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.344691    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.445634 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.445597    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.445829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.445682    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.445829 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.445711    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67nxm\" (UniqueName: \"kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.446396 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.446375    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.447975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.447954    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.472050 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.472016    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:14.502970 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.502941    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67nxm\" (UniqueName: \"kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm\") pod \"maas-api-revoke-sub-b53b0f97-7d6nc\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.513797 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.513775    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:14.623180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.623104    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:14.902563 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:14.902512    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"]
Nov 01 08:56:14.903359 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:56:14.903334    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdc4e30_398f_4f24_b754_7c97c0284877.slice/crio-298d24a758cc45607af30b4bcf975f518bf15b179b496cdc4a7db4f5ff298557 WatchSource:0}: Error finding container 298d24a758cc45607af30b4bcf975f518bf15b179b496cdc4a7db4f5ff298557: Status 404 returned error can't find the container with id 298d24a758cc45607af30b4bcf975f518bf15b179b496cdc4a7db4f5ff298557
Nov 01 08:56:15.251006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:15.250968    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc" event={"ID":"1fdc4e30-398f-4f24-b754-7c97c0284877","Type":"ContainerStarted","Data":"23d8bb97217c201fbe2fd9e02db226abd522d9a9fcafbc50418d1fd77cc40f71"}
Nov 01 08:56:15.251452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:15.251028    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc" event={"ID":"1fdc4e30-398f-4f24-b754-7c97c0284877","Type":"ContainerStarted","Data":"298d24a758cc45607af30b4bcf975f518bf15b179b496cdc4a7db4f5ff298557"}
Nov 01 08:56:15.360650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:15.360591    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc" podStartSLOduration=1.360577244 podStartE2EDuration="1.360577244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:15.358396557 +0000 UTC m=+1448.552791115" watchObservedRunningTime="2026-11-01 08:56:15.360577244 +0000 UTC m=+1448.554971836"
Nov 01 08:56:16.260081 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:16.260047    4314 generic.go:356] "Generic (PLEG): container finished" podID="1fdc4e30-398f-4f24-b754-7c97c0284877" containerID="23d8bb97217c201fbe2fd9e02db226abd522d9a9fcafbc50418d1fd77cc40f71" exitCode=0
Nov 01 08:56:16.260512 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:16.260135    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc" event={"ID":"1fdc4e30-398f-4f24-b754-7c97c0284877","Type":"ContainerDied","Data":"23d8bb97217c201fbe2fd9e02db226abd522d9a9fcafbc50418d1fd77cc40f71"}
Nov 01 08:56:17.434018 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.433993    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:17.483299 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.483270    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca\") pod \"1fdc4e30-398f-4f24-b754-7c97c0284877\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") "
Nov 01 08:56:17.483449 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.483312    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca\") pod \"1fdc4e30-398f-4f24-b754-7c97c0284877\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") "
Nov 01 08:56:17.483449 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.483419    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm\" (UniqueName: \"kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm\") pod \"1fdc4e30-398f-4f24-b754-7c97c0284877\" (UID: \"1fdc4e30-398f-4f24-b754-7c97c0284877\") "
Nov 01 08:56:17.483724 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.483693    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca" pod "1fdc4e30-398f-4f24-b754-7c97c0284877" (UID: "1fdc4e30-398f-4f24-b754-7c97c0284877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:56:17.484158 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.484136    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:56:17.485313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.485277    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca" pod "1fdc4e30-398f-4f24-b754-7c97c0284877" (UID: "1fdc4e30-398f-4f24-b754-7c97c0284877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:56:17.485432 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.485408    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm" pod "1fdc4e30-398f-4f24-b754-7c97c0284877" (UID: "1fdc4e30-398f-4f24-b754-7c97c0284877"). InnerVolumeSpecName "kube-api-access-67nxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:56:17.586327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.586243    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fdc4e30-398f-4f24-b754-7c97c0284877-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:56:17.586327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:17.586283    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67nxm\" (UniqueName: \"kubernetes.io/projected/1fdc4e30-398f-4f24-b754-7c97c0284877-kube-api-access-67nxm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:56:18.279584 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:18.279527    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"
Nov 01 08:56:18.279796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:18.279579    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc" event={"ID":"1fdc4e30-398f-4f24-b754-7c97c0284877","Type":"ContainerDied","Data":"298d24a758cc45607af30b4bcf975f518bf15b179b496cdc4a7db4f5ff298557"}
Nov 01 08:56:18.279796 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:18.279646    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298d24a758cc45607af30b4bcf975f518bf15b179b496cdc4a7db4f5ff298557"
Nov 01 08:56:19.302027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:19.301995    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:24.198695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:24.198660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:24.927567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:24.927500    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:29.098595 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:29.098559    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:29.286888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:29.286846    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:30.279131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:30.279096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:31.023271 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.023235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:31.298461 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.298387    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:31.508100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.508070    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"]
Nov 01 08:56:31.508718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.508695    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1fdc4e30-398f-4f24-b754-7c97c0284877" containerName="revoke-keys"
Nov 01 08:56:31.508718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.508713    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdc4e30-398f-4f24-b754-7c97c0284877" containerName="revoke-keys"
Nov 01 08:56:31.508877 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.508803    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdc4e30-398f-4f24-b754-7c97c0284877" containerName="revoke-keys"
Nov 01 08:56:31.512343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.512325    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.561419 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.561350    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8701a0-openshift-default-dockercfg-zmh2p\""
Nov 01 08:56:31.605669 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.605637    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:31.613765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.613736    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"]
Nov 01 08:56:31.672977 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.672938    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.672999    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673107    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673283 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673167    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673340 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673319    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673349    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673467 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673450    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673509 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673479    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwftq\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.673593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.673580    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.778556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.777647    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:31.778830 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.778798    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.779138 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.779112    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwftq\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.779359 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.779339    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.779579 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.779556    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.780504 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.779760    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.780727 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.780707    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.780843 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.780829    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.781053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.781038    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.781171 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.781156    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.782117 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.782094    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.784879 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.784852    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.788583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.785494    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.788583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.785766    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.788583 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.786863    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.804560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.804511    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.828788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.822032    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.844244 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.842524    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwftq\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:31.845119 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:31.845089    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token\") pod \"e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") " pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:32.127410 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:32.127330    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:32.551416 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:56:32.551384    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce33414_ccd3_4630_acc2_81c87015071a.slice/crio-7e10128ab1159b706040f2bd7ee8688eec9bbedf6bece0e24b982308a3d0ea60 WatchSource:0}: Error finding container 7e10128ab1159b706040f2bd7ee8688eec9bbedf6bece0e24b982308a3d0ea60: Status 404 returned error can't find the container with id 7e10128ab1159b706040f2bd7ee8688eec9bbedf6bece0e24b982308a3d0ea60
Nov 01 08:56:32.552157 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:32.552058    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"]
Nov 01 08:56:32.553995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:32.553965    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:56:32.554061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:32.554026    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:56:32.554061 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:32.554057    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:56:33.398383 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:33.398341    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" event={"ID":"9ce33414-ccd3-4630-acc2-81c87015071a","Type":"ContainerStarted","Data":"e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"}
Nov 01 08:56:33.398584 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:33.398408    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" event={"ID":"9ce33414-ccd3-4630-acc2-81c87015071a","Type":"ContainerStarted","Data":"7e10128ab1159b706040f2bd7ee8688eec9bbedf6bece0e24b982308a3d0ea60"}
Nov 01 08:56:33.559197 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:33.559143    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podStartSLOduration=2.559130724 podStartE2EDuration="2.559130724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:33.556628636 +0000 UTC m=+1466.751023203" watchObservedRunningTime="2026-11-01 08:56:33.559130724 +0000 UTC m=+1466.753525279"
Nov 01 08:56:34.127691 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:34.127660    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:34.132772 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:34.132736    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:34.405593 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:34.405505    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:34.406520 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:34.406499    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:56:34.415672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:34.415648    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:36.345526 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:36.345489    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:36.508067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:36.508019    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:36.662807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:36.662731    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:38.616482 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:38.616446    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:38.764963 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:38.764931    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:39.631123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:39.631089    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:39.828466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:39.828417    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:39.992178 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:39.989222    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:41.336300 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:56:41.336257    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:56:43.655557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:43.654627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:56:43.667554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:43.664623    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:56:43.686561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:43.686284    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:56:45.451837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:45.451794    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:50.411848 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:50.411765    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:54.826913 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:54.826876    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:56:55.182078 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:56:55.181996    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:00.002746 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:00.002706    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:04.818903 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:04.818861    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:08.237723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:08.237671    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:09.708816 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:09.708774    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:09.854318 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:09.854281    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:11.322053 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:57:11.322013    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:57:14.587608 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:14.587573    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:19.380883 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:19.380841    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:24.166560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:24.166500    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:24.321249 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:24.321217    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:24.475828 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:24.475742    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:26.740070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.740036    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"]
Nov 01 08:57:26.744631 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.744612    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.786452 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.786426    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"]
Nov 01 08:57:26.791099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.791077    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.841431 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.841370    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"]
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.887870    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.887932    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.887975    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888011    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888040    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888064    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8px\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888090    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888114    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888168    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888210    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888272    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnbv\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888318    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888392    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888429    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888548    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888624    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888669    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888699    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.889594 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.888818    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"]
Nov 01 08:57:26.990678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990583    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.990678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990649    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.990919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990758    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.990919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990827    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.990919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990869    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.990919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990901    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990953    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.990989    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991032    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991066    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991102    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991127    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8px\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991154    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991153    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991239 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991210    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991447    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991495    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991556    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991560    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991635    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991709    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnbv\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.991761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991750    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.992409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991778    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.992409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.991965    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.992409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.992184    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.992409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.992367    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.992780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.992443    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:26.993487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.993456    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:26.994888 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.994862    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:27.000006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.996518    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:27.000006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:26.999317    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:27.000835 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.000812    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:27.048324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.048257    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:27.048487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.048414    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:27.048487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.048434    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8px\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:27.048956 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.048932    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnbv\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:27.059726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.059702    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:27.105696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.105667    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:27.532738 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:57:27.532698    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0513ff0_437c_4ca8_9ec3_c0ee53bf9080.slice/crio-60ac5af3be3bc15df1cf5baee01a9f76296c69f4e45f526d93cfb03309bb9529 WatchSource:0}: Error finding container 60ac5af3be3bc15df1cf5baee01a9f76296c69f4e45f526d93cfb03309bb9529: Status 404 returned error can't find the container with id 60ac5af3be3bc15df1cf5baee01a9f76296c69f4e45f526d93cfb03309bb9529
Nov 01 08:57:27.534408 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.534384    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"]
Nov 01 08:57:27.535096 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.535060    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:27.535192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.535125    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:27.535192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.535153    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:27.615311 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:57:27.615280    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9358e53e_caf4_42da_be9c_2d18e49e5416.slice/crio-0e9251a437e3afab7af037efdd3c6a9df2e222891d6082f728cd20eed7c2add9 WatchSource:0}: Error finding container 0e9251a437e3afab7af037efdd3c6a9df2e222891d6082f728cd20eed7c2add9: Status 404 returned error can't find the container with id 0e9251a437e3afab7af037efdd3c6a9df2e222891d6082f728cd20eed7c2add9
Nov 01 08:57:27.617168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.617137    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"]
Nov 01 08:57:27.618415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.618374    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:27.618517 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.618464    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:27.618517 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.618508    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:27.908922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.908829    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" event={"ID":"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080","Type":"ContainerStarted","Data":"17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd"}
Nov 01 08:57:27.909389 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.908931    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" event={"ID":"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080","Type":"ContainerStarted","Data":"60ac5af3be3bc15df1cf5baee01a9f76296c69f4e45f526d93cfb03309bb9529"}
Nov 01 08:57:27.910949 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.910911    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" event={"ID":"9358e53e-caf4-42da-be9c-2d18e49e5416","Type":"ContainerStarted","Data":"1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405"}
Nov 01 08:57:27.911057 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:27.910964    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" event={"ID":"9358e53e-caf4-42da-be9c-2d18e49e5416","Type":"ContainerStarted","Data":"0e9251a437e3afab7af037efdd3c6a9df2e222891d6082f728cd20eed7c2add9"}
Nov 01 08:57:28.061797 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.060714    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:28.065572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.064747    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:28.065572 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.064807    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:28.105890 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.105854    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:28.107784 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.107756    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:28.107910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.107826    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:28.153124 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.153061    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podStartSLOduration=2.15304511 podStartE2EDuration="2.15304511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:28.027589709 +0000 UTC m=+1521.221984263" watchObservedRunningTime="2026-11-01 08:57:28.15304511 +0000 UTC m=+1521.347439665"
Nov 01 08:57:28.153524 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:28.153437    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podStartSLOduration=2.153429714 podStartE2EDuration="2.153429714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:28.152342004 +0000 UTC m=+1521.346736584" watchObservedRunningTime="2026-11-01 08:57:28.153429714 +0000 UTC m=+1521.347824268"
Nov 01 08:57:29.079556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:29.078929    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:29.079556 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:29.078999    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:29.107164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:29.107001    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:29.107164 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:29.107064    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:29.197570 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:29.193809    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:30.067358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:30.066900    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:30.067358 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:30.066974    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:30.107924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:30.107889    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:30.108331 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:30.107940    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:31.067557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:31.063833    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:31.067557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:31.063890    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:31.106885 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:31.106851    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:31.107108 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:31.107086    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:32.065559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:32.061909    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:32.065559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:32.061969    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:32.107666 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:32.106284    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:32.107666 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:32.106349    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:33.086558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:33.086036    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:33.086558 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:33.086117    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:33.110560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:33.108161    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:33.110560 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:33.108252    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:34.060958 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:34.060917    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 08:57:34.061137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:34.060993    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 08:57:34.106568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:34.106503    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 08:57:34.106961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:34.106602    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 08:57:34.215483 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:34.215445    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:35.427449 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.427381    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:35.550487 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.550449    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:35.551729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.551700    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:35.553725 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.553706    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 08:57:35.670910 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.670872    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:35.672053 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.672014    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:35.676274 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:35.676247    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 08:57:36.144469 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:36.144442    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:37.479851 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.479820    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"]
Nov 01 08:57:37.484549 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.484513    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.533278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.533248    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d0ba8f-openshift-default-dockercfg-kvclq\""
Nov 01 08:57:37.581837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.581801    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:37.585571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.585508    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"]
Nov 01 08:57:37.610945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.610913    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.610954    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611061    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611228 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611213    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611260    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611343 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611330    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611395 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611376    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611445    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kv5r\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.611505 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.611498    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713242 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713208    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713273    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713310    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713366    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713450 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713409    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kv5r\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713462    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713662    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713705    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713739    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713821 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713749    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713982 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713854    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.713982 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.713937    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.714187 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.714161    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.714250 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.714185    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.716761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.716737    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.716876 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.716794    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.777185 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.777081    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kv5r\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.777327 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.777293    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token\") pod \"e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") " pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:37.790455 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.790422    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:37.798834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:37.798785    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:38.235212 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:57:38.235179    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05119dac_77cf_47aa_ace8_b97cc03dfb1e.slice/crio-17389c3a8b1cb8ea83f8cc5a3bfc510167a153c38703b9e432497ae3d9eb31b9 WatchSource:0}: Error finding container 17389c3a8b1cb8ea83f8cc5a3bfc510167a153c38703b9e432497ae3d9eb31b9: Status 404 returned error can't find the container with id 17389c3a8b1cb8ea83f8cc5a3bfc510167a153c38703b9e432497ae3d9eb31b9
Nov 01 08:57:38.236132 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:38.236105    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"]
Nov 01 08:57:38.237768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:38.237728    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:38.237834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:38.237793    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:38.237834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:38.237822    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:39.000398 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:39.000363    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" event={"ID":"05119dac-77cf-47aa-ace8-b97cc03dfb1e","Type":"ContainerStarted","Data":"e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"}
Nov 01 08:57:39.000806 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:39.000417    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" event={"ID":"05119dac-77cf-47aa-ace8-b97cc03dfb1e","Type":"ContainerStarted","Data":"17389c3a8b1cb8ea83f8cc5a3bfc510167a153c38703b9e432497ae3d9eb31b9"}
Nov 01 08:57:39.186590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:39.184019    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podStartSLOduration=2.183995568 podStartE2EDuration="2.183995568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:39.173913247 +0000 UTC m=+1532.368307807" watchObservedRunningTime="2026-11-01 08:57:39.183995568 +0000 UTC m=+1532.378390124"
Nov 01 08:57:39.283712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:39.283620    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:39.799363 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:39.799326    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:39.805177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:39.805150    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:40.007762 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:40.007733    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:40.008732 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:40.008710    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:57:41.324458 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:57:41.324409    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:57:41.991959 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:41.991740    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:42.154118 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:42.154082    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:42.378612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:42.377022    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:44.048771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:44.048743    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"]
Nov 01 08:57:44.094978 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:44.094929    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad5da820-gn7tl"]
Nov 01 08:57:44.504567 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:44.504521    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:44.930814 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:44.930736    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617e5258-7bc4-4719-8d28-dca8ad3130d7" path="/var/lib/kubelet/pods/617e5258-7bc4-4719-8d28-dca8ad3130d7/volumes"
Nov 01 08:57:45.054218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:45.054185    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"]
Nov 01 08:57:45.101369 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:45.101326    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc086c9-rxzhk"]
Nov 01 08:57:45.143942 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:45.143911    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:45.321436 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:45.321389    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:46.929602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:46.929572    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f67cfc-ffe0-422b-b5de-02d9e758bd3f" path="/var/lib/kubelet/pods/72f67cfc-ffe0-422b-b5de-02d9e758bd3f/volumes"
Nov 01 08:57:48.085791 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:48.085757    4314 generic.go:356] "Generic (PLEG): container finished" podID="9ce33414-ccd3-4630-acc2-81c87015071a" containerID="e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd" exitCode=0
Nov 01 08:57:48.086318 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:48.085828    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" event={"ID":"9ce33414-ccd3-4630-acc2-81c87015071a","Type":"ContainerDied","Data":"e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"}
Nov 01 08:57:48.086397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:48.086377    4314 scope.go:122] "RemoveContainer" containerID="e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"
Nov 01 08:57:48.088258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:48.088222    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:48.088401 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:48.088306    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:48.088401 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:48.088339    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:49.096189 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:49.096141    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" event={"ID":"9ce33414-ccd3-4630-acc2-81c87015071a","Type":"ContainerStarted","Data":"059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55"}
Nov 01 08:57:49.127561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:49.127520    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:57:49.128324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:49.128284    4314 patch_prober.go:29] interesting pod/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 08:57:49.128415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:49.128342    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 08:57:49.405571 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:49.405442    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:57:49.682618 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:49.682196    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:50.128245 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:50.128211    4314 patch_prober.go:29] interesting pod/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 08:57:50.128668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:50.128265    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 08:57:51.128313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:51.128275    4314 patch_prober.go:29] interesting pod/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 08:57:51.128737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:51.128330    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 08:57:52.128130 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:52.128098    4314 patch_prober.go:29] interesting pod/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 08:57:52.128325 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:52.128155    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 08:57:53.128547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:53.128504    4314 patch_prober.go:29] interesting pod/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 08:57:53.128924 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:53.128583    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 08:57:54.049258 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.049223    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"]
Nov 01 08:57:54.095606 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.095572    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38cb09b-rgkdw"]
Nov 01 08:57:54.128054 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.128018    4314 patch_prober.go:29] interesting pod/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 08:57:54.128261 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.128075    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 08:57:54.210547 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.210495    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"]
Nov 01 08:57:54.215047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.215025    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.263473 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.263437    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:57:54.263678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.263599    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:57:54.263761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.263681    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:57:54.263761 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.263683    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:57:54.269033 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.269008    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"]
Nov 01 08:57:54.287326 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.287289    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2296\" (UniqueName: \"kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.287465 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.287359    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.287672 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.287642    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.389015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.388936    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.389168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.389101    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2296\" (UniqueName: \"kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.389168 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.389161    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.389693 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.389664    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.391561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.391512    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.448607 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.448572    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2296\" (UniqueName: \"kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296\") pod \"maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.513142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.513106    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:54.530346 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.530322    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:54.929659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.929631    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f83bcf-9efd-4f1f-b59f-9f818a031154" path="/var/lib/kubelet/pods/a2f83bcf-9efd-4f1f-b59f-9f818a031154/volumes"
Nov 01 08:57:54.937280 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:57:54.937250    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18813eb6_93ad_4ee9_b7d0_b17060a7218d.slice/crio-35689144b4b748e2576a41ef66119b723db71cc172dc67059b7642efa8e7ae2b WatchSource:0}: Error finding container 35689144b4b748e2576a41ef66119b723db71cc172dc67059b7642efa8e7ae2b: Status 404 returned error can't find the container with id 35689144b4b748e2576a41ef66119b723db71cc172dc67059b7642efa8e7ae2b
Nov 01 08:57:54.937636 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:54.937615    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"]
Nov 01 08:57:55.132266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:55.132235    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:57:55.133506 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:55.133483    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 08:57:55.140430 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:55.140408    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh" event={"ID":"18813eb6-93ad-4ee9-b7d0-b17060a7218d","Type":"ContainerStarted","Data":"35689144b4b748e2576a41ef66119b723db71cc172dc67059b7642efa8e7ae2b"}
Nov 01 08:57:55.771647 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:55.771615    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:56.152001 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.151905    4314 generic.go:356] "Generic (PLEG): container finished" podID="18813eb6-93ad-4ee9-b7d0-b17060a7218d" containerID="7e01712a2eaed4179612fc69bf3597057ceef98dc3578040a2aa27f84a844d3e" exitCode=0
Nov 01 08:57:56.152001 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.151988    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh" event={"ID":"18813eb6-93ad-4ee9-b7d0-b17060a7218d","Type":"ContainerDied","Data":"7e01712a2eaed4179612fc69bf3597057ceef98dc3578040a2aa27f84a844d3e"}
Nov 01 08:57:56.259800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.259759    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"]
Nov 01 08:57:56.295964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.293893    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.455616 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.451593    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-03832c-openshift-default-dockercfg-sf8zd\""
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.463944    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464050    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464146    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464200    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464232    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464347    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464383    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464478    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.465011 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.464562    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpbn\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.475055 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.475016    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:56.487852 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.487821    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"]
Nov 01 08:57:56.566300 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566259    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566365    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566417    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566508 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566444    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566548    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566589    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566675    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566738    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpbn\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.566868 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.566804    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.567101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.567077    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.567270 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.567170    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.567477 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.567431    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.567674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.567643    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.567754 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.567692    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.569889 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.569864    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.570204 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.570181    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.628702 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.627977    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.635765 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.635733    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpbn\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn\") pod \"e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") " pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:56.648282 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.648250    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:57:56.782405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:56.782376    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:57.205742 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:57:57.205697    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca0a0aa_ffe1_4ac5_b35d_326f3830df8d.slice/crio-d65193bd5942a987ed608554ee9e583df5fbb5ca1a75b918b14caee1b5381838 WatchSource:0}: Error finding container d65193bd5942a987ed608554ee9e583df5fbb5ca1a75b918b14caee1b5381838: Status 404 returned error can't find the container with id d65193bd5942a987ed608554ee9e583df5fbb5ca1a75b918b14caee1b5381838
Nov 01 08:57:57.206177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.206156    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"]
Nov 01 08:57:57.208409 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.208369    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:57.208526 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.208447    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:57.208526 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.208494    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:57:57.320598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.320570    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:57.376689 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.376655    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296\" (UniqueName: \"kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296\") pod \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") "
Nov 01 08:57:57.376922 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.376904    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca\") pod \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") "
Nov 01 08:57:57.377020 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.377002    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca\") pod \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\" (UID: \"18813eb6-93ad-4ee9-b7d0-b17060a7218d\") "
Nov 01 08:57:57.377332 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.377302    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca" pod "18813eb6-93ad-4ee9-b7d0-b17060a7218d" (UID: "18813eb6-93ad-4ee9-b7d0-b17060a7218d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:57:57.377665 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.377642    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:57:57.379364 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.379341    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca" pod "18813eb6-93ad-4ee9-b7d0-b17060a7218d" (UID: "18813eb6-93ad-4ee9-b7d0-b17060a7218d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:57:57.379456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.379400    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296" pod "18813eb6-93ad-4ee9-b7d0-b17060a7218d" (UID: "18813eb6-93ad-4ee9-b7d0-b17060a7218d"). InnerVolumeSpecName "kube-api-access-r2296". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:57:57.479166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.479079    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18813eb6-93ad-4ee9-b7d0-b17060a7218d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:57:57.479166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.479118    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2296\" (UniqueName: \"kubernetes.io/projected/18813eb6-93ad-4ee9-b7d0-b17060a7218d-kube-api-access-r2296\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:57:57.683372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.683337    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"]
Nov 01 08:57:57.684027 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.684010    4314 cpu_manager.go:414] "containerMap: removing container" podUID="18813eb6-93ad-4ee9-b7d0-b17060a7218d" containerName="revoke-keys"
Nov 01 08:57:57.684091 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.684028    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18813eb6-93ad-4ee9-b7d0-b17060a7218d" containerName="revoke-keys"
Nov 01 08:57:57.684161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.684148    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="18813eb6-93ad-4ee9-b7d0-b17060a7218d" containerName="revoke-keys"
Nov 01 08:57:57.688192 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.688174    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.739718 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.739688    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"]
Nov 01 08:57:57.783430 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.783393    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.783849 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.783625    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.783849 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.783732    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74q8\" (UniqueName: \"kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.885385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.885349    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.885589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.885466    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74q8\" (UniqueName: \"kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.885589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.885570    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.886132 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.886104    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.887840 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.887822    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:57.942776 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:57.942750    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74q8\" (UniqueName: \"kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8\") pod \"maas-api-revoke-sub-1e2af39e-cxx2q\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:58.004382 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.004304    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:57:58.168827 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.168791    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" event={"ID":"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d","Type":"ContainerStarted","Data":"7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"}
Nov 01 08:57:58.169012 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.168858    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" event={"ID":"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d","Type":"ContainerStarted","Data":"d65193bd5942a987ed608554ee9e583df5fbb5ca1a75b918b14caee1b5381838"}
Nov 01 08:57:58.170186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.170162    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"
Nov 01 08:57:58.170304 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.170200    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh" event={"ID":"18813eb6-93ad-4ee9-b7d0-b17060a7218d","Type":"ContainerDied","Data":"35689144b4b748e2576a41ef66119b723db71cc172dc67059b7642efa8e7ae2b"}
Nov 01 08:57:58.170304 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.170239    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35689144b4b748e2576a41ef66119b723db71cc172dc67059b7642efa8e7ae2b"
Nov 01 08:57:58.286080 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.285975    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podStartSLOduration=2.285961985 podStartE2EDuration="2.285961985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:58.281166826 +0000 UTC m=+1551.475561383" watchObservedRunningTime="2026-11-01 08:57:58.285961985 +0000 UTC m=+1551.480356540"
Nov 01 08:57:58.376320 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:57:58.376280    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef04bc2d_eb57_44be_80e6_26f98467f4a6.slice/crio-45c75919a1179a96b9dd796a52d2370ac03c7ba1aa6fabf50ef89ccc854f72bc WatchSource:0}: Error finding container 45c75919a1179a96b9dd796a52d2370ac03c7ba1aa6fabf50ef89ccc854f72bc: Status 404 returned error can't find the container with id 45c75919a1179a96b9dd796a52d2370ac03c7ba1aa6fabf50ef89ccc854f72bc
Nov 01 08:57:58.376946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.376857    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"]
Nov 01 08:57:58.783605 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.783566    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:58.788904 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:58.788853    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:59.180811 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:59.180767    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q" event={"ID":"ef04bc2d-eb57-44be-80e6-26f98467f4a6","Type":"ContainerStarted","Data":"9bfa5e5f12ac816dc104e4968114fdab24fd53553f431f1b76576e09bc8dea52"}
Nov 01 08:57:59.181021 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:59.180831    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q" event={"ID":"ef04bc2d-eb57-44be-80e6-26f98467f4a6","Type":"ContainerStarted","Data":"45c75919a1179a96b9dd796a52d2370ac03c7ba1aa6fabf50ef89ccc854f72bc"}
Nov 01 08:57:59.181106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:59.181080    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:59.182015 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:59.181996    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:57:59.292298 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:59.292242    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q" podStartSLOduration=2.292228909 podStartE2EDuration="2.292228909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:59.288310694 +0000 UTC m=+1552.482705243" watchObservedRunningTime="2026-11-01 08:57:59.292228909 +0000 UTC m=+1552.486623464"
Nov 01 08:57:59.782338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:57:59.782307    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:00.191506 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:00.191406    4314 generic.go:356] "Generic (PLEG): container finished" podID="ef04bc2d-eb57-44be-80e6-26f98467f4a6" containerID="9bfa5e5f12ac816dc104e4968114fdab24fd53553f431f1b76576e09bc8dea52" exitCode=0
Nov 01 08:58:00.191999 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:00.191499    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q" event={"ID":"ef04bc2d-eb57-44be-80e6-26f98467f4a6","Type":"ContainerDied","Data":"9bfa5e5f12ac816dc104e4968114fdab24fd53553f431f1b76576e09bc8dea52"}
Nov 01 08:58:01.374800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.374776    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:58:01.415674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.415635    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:01.447651 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.447627    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca\") pod \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") "
Nov 01 08:58:01.447808 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.447764    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca\") pod \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") "
Nov 01 08:58:01.447907 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.447891    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8\" (UniqueName: \"kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8\") pod \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\" (UID: \"ef04bc2d-eb57-44be-80e6-26f98467f4a6\") "
Nov 01 08:58:01.448045 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.448005    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca" pod "ef04bc2d-eb57-44be-80e6-26f98467f4a6" (UID: "ef04bc2d-eb57-44be-80e6-26f98467f4a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:58:01.448385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.448364    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:01.449921 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.449878    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca" pod "ef04bc2d-eb57-44be-80e6-26f98467f4a6" (UID: "ef04bc2d-eb57-44be-80e6-26f98467f4a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:58:01.449992 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.449977    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8" pod "ef04bc2d-eb57-44be-80e6-26f98467f4a6" (UID: "ef04bc2d-eb57-44be-80e6-26f98467f4a6"). InnerVolumeSpecName "kube-api-access-q74q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:58:01.550004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.549972    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q74q8\" (UniqueName: \"kubernetes.io/projected/ef04bc2d-eb57-44be-80e6-26f98467f4a6-kube-api-access-q74q8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:01.550004 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.550000    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef04bc2d-eb57-44be-80e6-26f98467f4a6-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:01.594586 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.594508    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:01.744813 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:01.744780    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:02.208035 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:02.207955    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"
Nov 01 08:58:02.208035 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:02.207990    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q" event={"ID":"ef04bc2d-eb57-44be-80e6-26f98467f4a6","Type":"ContainerDied","Data":"45c75919a1179a96b9dd796a52d2370ac03c7ba1aa6fabf50ef89ccc854f72bc"}
Nov 01 08:58:02.208218 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:02.208037    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c75919a1179a96b9dd796a52d2370ac03c7ba1aa6fabf50ef89ccc854f72bc"
Nov 01 08:58:04.512493 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:04.512457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:04.696280 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:04.695268    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:04.972695 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:04.972610    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:05.065614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.065581    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"]
Nov 01 08:58:05.101486 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.101454    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93423d91-znnzs"]
Nov 01 08:58:05.106087 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.106057    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"]
Nov 01 08:58:05.106700 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.106683    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ef04bc2d-eb57-44be-80e6-26f98467f4a6" containerName="revoke-keys"
Nov 01 08:58:05.106770 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.106701    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef04bc2d-eb57-44be-80e6-26f98467f4a6" containerName="revoke-keys"
Nov 01 08:58:05.106838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.106827    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef04bc2d-eb57-44be-80e6-26f98467f4a6" containerName="revoke-keys"
Nov 01 08:58:05.110831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.110808    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.141329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.141305    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"]
Nov 01 08:58:05.181742 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.181712    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-03832c-dockercfg-vdbz6\""
Nov 01 08:58:05.196169 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.196145    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-03832c\""
Nov 01 08:58:05.211083 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.211041    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") " pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.211255 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.211149    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5mf\" (UniqueName: \"kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf\") pod \"payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") " pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.312802 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.312760    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") " pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.312989 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.312828    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5mf\" (UniqueName: \"kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf\") pod \"payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") " pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.314451 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.314421    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") " pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.370443 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.370410    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5mf\" (UniqueName: \"kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf\") pod \"payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") " pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.425094 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.425062    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:05.742454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.742410    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"]
Nov 01 08:58:05.800694 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.800664    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"]
Nov 01 08:58:05.842612 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:05.842586    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"]
Nov 01 08:58:06.097884 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.097771    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:06.248320 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.248253    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx" event={"ID":"b1bbc18c-3927-4686-9faf-5147d4ccb398","Type":"ContainerStarted","Data":"cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54"}
Nov 01 08:58:06.248523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.248330    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx" event={"ID":"b1bbc18c-3927-4686-9faf-5147d4ccb398","Type":"ContainerStarted","Data":"156b857ca0edcef008aa84b2f7cbe3c521bbc3fdaa8554c8656dcd4ccab36c1f"}
Nov 01 08:58:06.248523 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.248413    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:06.264697 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.264657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:06.365376 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.365257    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx" podStartSLOduration=1.365236111 podStartE2EDuration="1.365236111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:06.355867895 +0000 UTC m=+1559.550262453" watchObservedRunningTime="2026-11-01 08:58:06.365236111 +0000 UTC m=+1559.559630668"
Nov 01 08:58:06.929449 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:06.929420    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4" path="/var/lib/kubelet/pods/8f4ba9fd-db71-46d8-b5b8-d0429aab0fd4/volumes"
Nov 01 08:58:07.047375 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:07.047343    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"]
Nov 01 08:58:07.051737 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:07.051709    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7175babb-7kdn7"]
Nov 01 08:58:08.726557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:08.726030    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:08.937341 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:08.936817    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6" path="/var/lib/kubelet/pods/27d368aa-846b-4a0a-9dc7-7aab4d5d7eb6/volumes"
Nov 01 08:58:10.311166 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:10.311131    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:10.743264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:10.743227    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerName="payload-processing" containerID="cri-o://d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411" gracePeriod=25
Nov 01 08:58:10.843561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:10.843505    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerName="payload-pre-processing" containerID="cri-o://ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00" gracePeriod=25
Nov 01 08:58:11.047404 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:11.047326    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"]
Nov 01 08:58:11.093226 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:11.093195    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa904f3-7hzt5"]
Nov 01 08:58:11.323038 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:58:11.322953    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:58:12.937455 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:12.937418    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d098af-f183-4003-9f1f-481b29f10257" path="/var/lib/kubelet/pods/f5d098af-f183-4003-9f1f-481b29f10257/volumes"
Nov 01 08:58:15.078575 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:15.078513    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:16.394602 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:16.394562    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 08:58:16.395101 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:16.394592    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 08:58:17.300125 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:17.300093    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 08:58:19.917296 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:19.917216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:23.332047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:23.332015    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:25.117859 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:25.117823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:25.262717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:25.262660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:26.394650 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:26.394614    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 08:58:26.395212 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:26.394614    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 08:58:27.047180 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:27.047144    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"]
Nov 01 08:58:27.093373 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:27.093340    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af00103-txhqr"]
Nov 01 08:58:28.929738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:28.929705    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38d11b1-bfd0-45a9-ad5e-be220f986cab" path="/var/lib/kubelet/pods/a38d11b1-bfd0-45a9-ad5e-be220f986cab/volumes"
Nov 01 08:58:29.056723 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:29.056692    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"]
Nov 01 08:58:29.103006 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:29.102976    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4797ed5c-md8qv"]
Nov 01 08:58:30.384972 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:30.384914    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:30.938674 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:30.938626    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb98d16-d7f3-48c6-af69-fa394ec87309" path="/var/lib/kubelet/pods/afb98d16-d7f3-48c6-af69-fa394ec87309/volumes"
Nov 01 08:58:31.022814 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.022786    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:58:31.084153 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.084116    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx\" (UniqueName: \"kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx\") pod \"2c136930-d835-4535-b336-9ebae7bb8fe7\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") "
Nov 01 08:58:31.084415 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.084395    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume\") pod \"2c136930-d835-4535-b336-9ebae7bb8fe7\" (UID: \"2c136930-d835-4535-b336-9ebae7bb8fe7\") "
Nov 01 08:58:31.084807 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.084779    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume" pod "2c136930-d835-4535-b336-9ebae7bb8fe7" (UID: "2c136930-d835-4535-b336-9ebae7bb8fe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:58:31.085145 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.085124    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c136930-d835-4535-b336-9ebae7bb8fe7-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:31.086266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.086243    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx" pod "2c136930-d835-4535-b336-9ebae7bb8fe7" (UID: "2c136930-d835-4535-b336-9ebae7bb8fe7"). InnerVolumeSpecName "kube-api-access-s89kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:58:31.106990 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.106969    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:58:31.185780 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.185750    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume\") pod \"6b3e831d-b67a-48d8-b593-23274bcf3630\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") "
Nov 01 08:58:31.185961 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.185939    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm\" (UniqueName: \"kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm\") pod \"6b3e831d-b67a-48d8-b593-23274bcf3630\" (UID: \"6b3e831d-b67a-48d8-b593-23274bcf3630\") "
Nov 01 08:58:31.186214 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.186183    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume" pod "6b3e831d-b67a-48d8-b593-23274bcf3630" (UID: "6b3e831d-b67a-48d8-b593-23274bcf3630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:58:31.186483 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.186464    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s89kx\" (UniqueName: \"kubernetes.io/projected/2c136930-d835-4535-b336-9ebae7bb8fe7-kube-api-access-s89kx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:31.186592 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.186486    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3e831d-b67a-48d8-b593-23274bcf3630-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:31.188028 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.188007    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm" pod "6b3e831d-b67a-48d8-b593-23274bcf3630" (UID: "6b3e831d-b67a-48d8-b593-23274bcf3630"). InnerVolumeSpecName "kube-api-access-vn7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:58:31.288561 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.288475    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7xm\" (UniqueName: \"kubernetes.io/projected/6b3e831d-b67a-48d8-b593-23274bcf3630-kube-api-access-vn7xm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:31.311451 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.311412    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:31.516289 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.516254    4314 generic.go:356] "Generic (PLEG): container finished" podID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerID="d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411" exitCode=0
Nov 01 08:58:31.516787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.516309    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"
Nov 01 08:58:31.516787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.516340    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" event={"ID":"2c136930-d835-4535-b336-9ebae7bb8fe7","Type":"ContainerDied","Data":"d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411"}
Nov 01 08:58:31.516787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.516407    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz" event={"ID":"2c136930-d835-4535-b336-9ebae7bb8fe7","Type":"ContainerDied","Data":"5492453208b5dca4c7180de364ee6d858f73d26ab237ea0bb2ea527562689c0d"}
Nov 01 08:58:31.516787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.516441    4314 scope.go:122] "RemoveContainer" containerID="d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411"
Nov 01 08:58:31.518030 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.518007    4314 generic.go:356] "Generic (PLEG): container finished" podID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerID="ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00" exitCode=0
Nov 01 08:58:31.518146 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.518059    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"
Nov 01 08:58:31.518146 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.518098    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" event={"ID":"6b3e831d-b67a-48d8-b593-23274bcf3630","Type":"ContainerDied","Data":"ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00"}
Nov 01 08:58:31.518229 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.518150    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp" event={"ID":"6b3e831d-b67a-48d8-b593-23274bcf3630","Type":"ContainerDied","Data":"16a7d96378a1be6658a48fc6d618e98db69a9896f3664479f5ea49078bea0567"}
Nov 01 08:58:31.527251 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.527187    4314 scope.go:122] "RemoveContainer" containerID="d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411"
Nov 01 08:58:31.527630 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:58:31.527522    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411\": container with ID starting with d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411 not found: ID does not exist" containerID="d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411"
Nov 01 08:58:31.527630 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.527575    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411"} err="failed to get container status \"d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411\": rpc error: code = NotFound desc = could not find container \"d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411\": container with ID starting with d2e35f28f5d64e1ff8aed98b70a221068586615b889721ba0d6a9fe82a0ce411 not found: ID does not exist"
Nov 01 08:58:31.527630 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.527598    4314 scope.go:122] "RemoveContainer" containerID="ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00"
Nov 01 08:58:31.540414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.540398    4314 scope.go:122] "RemoveContainer" containerID="ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00"
Nov 01 08:58:31.540881 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:58:31.540857    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00\": container with ID starting with ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00 not found: ID does not exist" containerID="ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00"
Nov 01 08:58:31.540948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.540890    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00"} err="failed to get container status \"ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00\": rpc error: code = NotFound desc = could not find container \"ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00\": container with ID starting with ba450f7a3f9fa897b79e13ec885e6e655f130ff0874b33703e6ce0015be1ff00 not found: ID does not exist"
Nov 01 08:58:31.681064 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.681025    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"]
Nov 01 08:58:31.683539 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.683510    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-z9pqp"]
Nov 01 08:58:31.777698 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.777667    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"]
Nov 01 08:58:31.783252 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:31.783228    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-q5mnz"]
Nov 01 08:58:32.332900 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:32.332867    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:32.478466 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:32.478430    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:32.929063 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:32.929030    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" path="/var/lib/kubelet/pods/2c136930-d835-4535-b336-9ebae7bb8fe7/volumes"
Nov 01 08:58:32.929467 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:32.929452    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" path="/var/lib/kubelet/pods/6b3e831d-b67a-48d8-b593-23274bcf3630/volumes"
Nov 01 08:58:35.555414 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:35.555379    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:35.751568 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:35.748090    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:35.951350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:35.951263    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:36.262756 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:36.262716    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:37.153195 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153161    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"]
Nov 01 08:58:37.153834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153816    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerName="payload-processing"
Nov 01 08:58:37.153834 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153834    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerName="payload-processing"
Nov 01 08:58:37.153926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153865    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerName="payload-pre-processing"
Nov 01 08:58:37.153926 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153870    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerName="payload-pre-processing"
Nov 01 08:58:37.153995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153953    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c136930-d835-4535-b336-9ebae7bb8fe7" containerName="payload-processing"
Nov 01 08:58:37.153995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.153962    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3e831d-b67a-48d8-b593-23274bcf3630" containerName="payload-pre-processing"
Nov 01 08:58:37.158729 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.158707    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186559 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186517    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186673 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186570    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186689    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186753 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186731    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186826 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186814    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186859    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvjj\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186921 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186907    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.186956 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.186936    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.187048 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.187030    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.261914 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.261876    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"]
Nov 01 08:58:37.263514 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.263481    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:37.288585 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288513    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288602    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288738 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288698    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288742    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288856    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288890    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvjj\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.288975 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288961    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.289140 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.288999    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.289140 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.289033    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.289288 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.289224    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.289454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.289434    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.289890 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.289867    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.290137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.290115    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.291790 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.291749    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.292457 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.292410    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.295099 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.295078    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3901b8-openshift-default-dockercfg-xqhg4\""
Nov 01 08:58:37.350123 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.350080    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.350350 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.350117    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvjj\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj\") pod \"e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") " pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.472236 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.472140    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:37.561617 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.561583    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:37.898420 ip-10-0-134-93 kubenswrapper[4314]: W1101 08:58:37.898389    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0673bae6_a048_4b51_8444_988db5035dec.slice/crio-8b3825b2fe2e8d10fb9b0c17939f3c12916060d44f17ad23504e2581827f530e WatchSource:0}: Error finding container 8b3825b2fe2e8d10fb9b0c17939f3c12916060d44f17ad23504e2581827f530e: Status 404 returned error can't find the container with id 8b3825b2fe2e8d10fb9b0c17939f3c12916060d44f17ad23504e2581827f530e
Nov 01 08:58:37.898591 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.898570    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"]
Nov 01 08:58:37.900875 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.900839    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:58:37.900967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.900909    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:58:37.900967 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.900955    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:58:37.911025 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:37.911003    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:38.611852 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:38.611812    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" event={"ID":"0673bae6-a048-4b51-8444-988db5035dec","Type":"ContainerStarted","Data":"3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"}
Nov 01 08:58:38.612302 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:38.611880    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" event={"ID":"0673bae6-a048-4b51-8444-988db5035dec","Type":"ContainerStarted","Data":"8b3825b2fe2e8d10fb9b0c17939f3c12916060d44f17ad23504e2581827f530e"}
Nov 01 08:58:38.822349 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:38.822288    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podStartSLOduration=1.822272844 podStartE2EDuration="1.822272844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:38.775226823 +0000 UTC m=+1591.969621370" watchObservedRunningTime="2026-11-01 08:58:38.822272844 +0000 UTC m=+1592.016667398"
Nov 01 08:58:39.472856 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:39.472817    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:39.478586 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:39.478559    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:39.621266 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:39.621235    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:39.622043 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:39.622024    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:58:40.952289 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:40.952260    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:41.049338 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:41.049304    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"]
Nov 01 08:58:41.093749 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:41.093718    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003cca03-kkv29"]
Nov 01 08:58:41.323394 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:58:41.323359    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:58:41.514874 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:41.514843    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:41.688747 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:41.688670    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:41.848820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:41.848787    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:42.933730 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:42.931879    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17124d9e-fb64-45bc-8e07-a804a8e09456" path="/var/lib/kubelet/pods/17124d9e-fb64-45bc-8e07-a804a8e09456/volumes"
Nov 01 08:58:45.532787 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.532749    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"]
Nov 01 08:58:45.540142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.540114    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.589768 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.589738    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"]
Nov 01 08:58:45.608176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.608143    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.608456 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.608432    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.608614 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.608590    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9p7\" (UniqueName: \"kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.630464 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.630409    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:58:45.630659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.630486    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 08:58:45.630659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.630606    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:58:45.630831 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.630676    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:58:45.709904 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.709862    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.710070 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.709944    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.710112 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.710076    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9p7\" (UniqueName: \"kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.710574 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.710552    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.712163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.712141    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.767272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.767241    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9p7\" (UniqueName: \"kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7\") pod \"maas-api-revoke-sub-c4a28168-7fwxg\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:45.854529 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:45.854464    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:46.020716 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.020688    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:46.045946 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.045918    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"]
Nov 01 08:58:46.050148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.050123    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b329ab73-7pfsr"]
Nov 01 08:58:46.201995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.199007    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:46.255611 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.255565    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"]
Nov 01 08:58:46.371964 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.371860    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:46.725102 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.725008    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg" event={"ID":"9109b660-10be-47f2-a6f6-d39ebc028aed","Type":"ContainerStarted","Data":"520d1e59522d48439425e57c662c197ef977ebdf205f56a3851f7471032c9d5d"}
Nov 01 08:58:46.941161 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:46.941070    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf60a7d-b6b0-442e-bee1-a790ac5f0748" path="/var/lib/kubelet/pods/eaf60a7d-b6b0-442e-bee1-a790ac5f0748/volumes"
Nov 01 08:58:47.734176 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:47.734146    4314 generic.go:356] "Generic (PLEG): container finished" podID="9109b660-10be-47f2-a6f6-d39ebc028aed" containerID="f764643204b3abd41554a01a0f1719fab61b10a7a2f24289c12562a6bab775fe" exitCode=0
Nov 01 08:58:47.734518 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:47.734225    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg" event={"ID":"9109b660-10be-47f2-a6f6-d39ebc028aed","Type":"ContainerDied","Data":"f764643204b3abd41554a01a0f1719fab61b10a7a2f24289c12562a6bab775fe"}
Nov 01 08:58:48.904316 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:48.904285    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:49.089958 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.089878    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca\") pod \"9109b660-10be-47f2-a6f6-d39ebc028aed\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") "
Nov 01 08:58:49.090135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.089967    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7\" (UniqueName: \"kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7\") pod \"9109b660-10be-47f2-a6f6-d39ebc028aed\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") "
Nov 01 08:58:49.090135 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.090102    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca\") pod \"9109b660-10be-47f2-a6f6-d39ebc028aed\" (UID: \"9109b660-10be-47f2-a6f6-d39ebc028aed\") "
Nov 01 08:58:49.090272 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.090248    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca" pod "9109b660-10be-47f2-a6f6-d39ebc028aed" (UID: "9109b660-10be-47f2-a6f6-d39ebc028aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:58:49.090836 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.090811    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:49.092069 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.092049    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7" pod "9109b660-10be-47f2-a6f6-d39ebc028aed" (UID: "9109b660-10be-47f2-a6f6-d39ebc028aed"). InnerVolumeSpecName "kube-api-access-tf9p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:58:49.092163 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.092145    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca" pod "9109b660-10be-47f2-a6f6-d39ebc028aed" (UID: "9109b660-10be-47f2-a6f6-d39ebc028aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:58:49.192320 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.192286    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9p7\" (UniqueName: \"kubernetes.io/projected/9109b660-10be-47f2-a6f6-d39ebc028aed-kube-api-access-tf9p7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:49.192320 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.192319    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9109b660-10be-47f2-a6f6-d39ebc028aed-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:49.755243 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.755214    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"
Nov 01 08:58:49.755405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.755264    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg" event={"ID":"9109b660-10be-47f2-a6f6-d39ebc028aed","Type":"ContainerDied","Data":"520d1e59522d48439425e57c662c197ef977ebdf205f56a3851f7471032c9d5d"}
Nov 01 08:58:49.755405 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:49.755316    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="520d1e59522d48439425e57c662c197ef977ebdf205f56a3851f7471032c9d5d"
Nov 01 08:58:51.216872 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:51.216830    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:51.384676 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:51.384643    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:51.534712 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:51.534684    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:55.567590 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:55.567524    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:57.376137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:57.376074    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:58:59.062142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:59.062097    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"]
Nov 01 08:58:59.106945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:58:59.106907    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511c8177-mptbg"]
Nov 01 08:59:00.929522 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:00.929488    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa149c0-c9a7-4f8b-b2a2-b42c264ba41e" path="/var/lib/kubelet/pods/afa149c0-c9a7-4f8b-b2a2-b42c264ba41e/volumes"
Nov 01 08:59:02.017060 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:02.017020    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:02.053488 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:02.053461    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"]
Nov 01 08:59:02.107244 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:02.107211    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-717ab89d-j6grf"]
Nov 01 08:59:02.158584 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:02.158549    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:02.382177 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:02.382139    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:02.928820 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:02.928781    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b8240b-c0c8-4d14-8421-5a84f1caea3d" path="/var/lib/kubelet/pods/d8b8240b-c0c8-4d14-8421-5a84f1caea3d/volumes"
Nov 01 08:59:03.534878 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:03.534843    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:04.047736 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:04.047706    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"]
Nov 01 08:59:04.048105 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:04.048083    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:04.093142 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:04.093108    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94afd23a-l26kt"]
Nov 01 08:59:04.830067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:04.830036    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:04.929837 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:04.929804    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642b038d-db97-4f6b-b976-191dd2a175ea" path="/var/lib/kubelet/pods/642b038d-db97-4f6b-b976-191dd2a175ea/volumes"
Nov 01 08:59:07.465141 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:07.465108    4314 scope.go:122] "RemoveContainer" containerID="685e2234a02604b25878da7f995bb0ed2ca1225fe7bbdf97b1e5416b7551e98f"
Nov 01 08:59:07.478103 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:07.478084    4314 scope.go:122] "RemoveContainer" containerID="bf9f654b3f3c87e441633d8e524ff1d62d4cfda76082c79af19f0daba65859b1"
Nov 01 08:59:07.490304 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:07.490284    4314 scope.go:122] "RemoveContainer" containerID="26c3f35b7cfa5ce56762eb3d4f82bc09a0c7a346b7cca7a2ca2bc4c67ccf341b"
Nov 01 08:59:07.501305 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:07.501287    4314 scope.go:122] "RemoveContainer" containerID="2e319a266932d65e54987ad73ff1a8c393924900a8347d9448b63ab6261070d4"
Nov 01 08:59:07.509871 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:07.509852    4314 scope.go:122] "RemoveContainer" containerID="12e6e157d509dc66b66a01b094cd96de1aa22c30ace713aebbfdf65874166dcb"
Nov 01 08:59:07.611237 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:07.611201    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:09.067786 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.067750    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:09.941309 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.941278    4314 generic.go:356] "Generic (PLEG): container finished" podID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerID="e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6" exitCode=0
Nov 01 08:59:09.941589 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.941356    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" event={"ID":"05119dac-77cf-47aa-ace8-b97cc03dfb1e","Type":"ContainerDied","Data":"e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"}
Nov 01 08:59:09.941917 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.941899    4314 scope.go:122] "RemoveContainer" containerID="e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"
Nov 01 08:59:09.943566 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.943517    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:09.943659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.943606    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:09.943659 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:09.943645    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:10.008291 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:10.008265    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:59:10.849625 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:10.849553    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:10.950131 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:10.950090    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" event={"ID":"05119dac-77cf-47aa-ace8-b97cc03dfb1e","Type":"ContainerStarted","Data":"2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0"}
Nov 01 08:59:11.325010 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:59:11.324961    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:59:11.798705 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:11.798660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:11.798939 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:11.798918    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:59:11.799800 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:11.799778    4314 patch_prober.go:29] interesting pod/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 08:59:11.799905 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:11.799822    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 08:59:11.948499 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:11.946866    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:12.801278 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:12.801239    4314 patch_prober.go:29] interesting pod/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 08:59:12.801632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:12.801596    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 08:59:12.981132 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:12.981096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:13.799137 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:13.799103    4314 patch_prober.go:29] interesting pod/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 08:59:13.799313 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:13.799153    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 08:59:14.799106 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:14.799071    4314 patch_prober.go:29] interesting pod/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 08:59:14.799496 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:14.799130    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 08:59:15.047289 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:15.047254    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"]
Nov 01 08:59:15.093711 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:15.093622    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef83188e-87pft"]
Nov 01 08:59:15.799233 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:15.799195    4314 patch_prober.go:29] interesting pod/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 08:59:15.799653 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:15.799275    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 08:59:16.046139 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.046105    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"]
Nov 01 08:59:16.092726 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.092601    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daa0279-2f5mc"]
Nov 01 08:59:16.543944 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.543910    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:16.804202 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.804129    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:59:16.804656 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.804639    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:59:16.805372 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.805349    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 08:59:16.929155 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.929122    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b422f0d6-af8e-444d-acfc-cc1925db1fc2" path="/var/lib/kubelet/pods/b422f0d6-af8e-444d-acfc-cc1925db1fc2/volumes"
Nov 01 08:59:16.929622 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:16.929605    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9" path="/var/lib/kubelet/pods/d5438ec9-3a7a-49c7-a6d6-fc8cb699e8f9/volumes"
Nov 01 08:59:17.047925 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:17.047895    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"]
Nov 01 08:59:17.052988 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:17.052963    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7302c-wnckc"]
Nov 01 08:59:18.173675 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:18.173641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:18.929717 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:18.929687    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb74d753-b538-4b8b-8075-8815d0674ba6" path="/var/lib/kubelet/pods/eb74d753-b538-4b8b-8075-8815d0674ba6/volumes"
Nov 01 08:59:23.021833 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:23.021804    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:27.914392 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:27.914352    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:32.637361 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:32.637331    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:32.790783 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:32.790747    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:32.937486 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:32.937397    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:34.296256 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:34.296219    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:35.943189 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:35.943117    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:36.197696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:36.197659    4314 generic.go:356] "Generic (PLEG): container finished" podID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerID="7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe" exitCode=0
Nov 01 08:59:36.197894 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:36.197705    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" event={"ID":"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d","Type":"ContainerDied","Data":"7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"}
Nov 01 08:59:36.198172 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:36.198151    4314 scope.go:122] "RemoveContainer" containerID="7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"
Nov 01 08:59:36.199838 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:36.199805    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:36.199945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:36.199888    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:36.199945 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:36.199923    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:37.048210 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.048179    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"]
Nov 01 08:59:37.052046 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.052026    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bdb9f64-m9mwg"]
Nov 01 08:59:37.206315 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.206280    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" event={"ID":"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d","Type":"ContainerStarted","Data":"fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76"}
Nov 01 08:59:37.782755 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.782722    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:59:37.783072 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.782872    4314 patch_prober.go:29] interesting pod/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 08:59:37.783170 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.783105    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 08:59:37.845948 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:37.845905    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:38.782891 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:38.782854    4314 patch_prober.go:29] interesting pod/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 08:59:38.783397 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:38.782922    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 08:59:38.929696 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:38.929664    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44dd4db2-d800-4158-940c-903b2c8f591e" path="/var/lib/kubelet/pods/44dd4db2-d800-4158-940c-903b2c8f591e/volumes"
Nov 01 08:59:39.782902 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:39.782861    4314 patch_prober.go:29] interesting pod/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 08:59:39.783402 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:39.782937    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 08:59:40.782919 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:40.782883    4314 patch_prober.go:29] interesting pod/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 08:59:40.783310 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:40.782941    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 08:59:41.323759 ip-10-0-134-93 kubenswrapper[4314]: E1101 08:59:41.323726    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 08:59:41.782943 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:41.782905    4314 patch_prober.go:29] interesting pod/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 08:59:41.783385 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:41.782987    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 08:59:41.826598 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:41.826569    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:42.080763 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:42.080625    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:42.747455 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:42.747418    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:42.802789 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:42.802758    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:59:42.804047 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:42.804025    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:59:42.805699 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:42.805674    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 08:59:42.892788 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:42.892726    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:43.744983 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:43.744953    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:44.030577 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.030469    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:44.058678 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.058641    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"]
Nov 01 08:59:44.095186 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.095156    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b4bdcd1-l92d4"]
Nov 01 08:59:44.183668 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.183608    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:44.341303 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.341205    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"]
Nov 01 08:59:44.343175 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.343126    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d31acd45-7qwjh"]
Nov 01 08:59:44.929301 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.929266    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18813eb6-93ad-4ee9-b7d0-b17060a7218d" path="/var/lib/kubelet/pods/18813eb6-93ad-4ee9-b7d0-b17060a7218d/volumes"
Nov 01 08:59:44.929789 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:44.929772    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b7ad41-635e-497b-a9d5-ba1294ee0fff" path="/var/lib/kubelet/pods/c7b7ad41-635e-497b-a9d5-ba1294ee0fff/volumes"
Nov 01 08:59:47.493554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:47.493501    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:48.254348 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:48.254292    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:49.282557 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:49.279783    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:50.550092 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:50.550059    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:50.696148 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:50.696118    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:51.763079 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:51.763041    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:51.932067 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:51.932036    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:53.995480 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:53.995449    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:54.191410 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:54.191370    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:54.676847 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:54.676800    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:55.330454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.330160    4314 patch_prober.go:29] interesting pod/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": read tcp 10.132.0.2:59230->10.132.0.125:15021: read: connection reset by peer" start-of-body=
Nov 01 08:59:55.330454 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.330224    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": read tcp 10.132.0.2:59230->10.132.0.125:15021: read: connection reset by peer"
Nov 01 08:59:55.396626 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.396598    4314 generic.go:356] "Generic (PLEG): container finished" podID="0673bae6-a048-4b51-8444-988db5035dec" containerID="3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1" exitCode=0
Nov 01 08:59:55.396757 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.396636    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" event={"ID":"0673bae6-a048-4b51-8444-988db5035dec","Type":"ContainerDied","Data":"3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"}
Nov 01 08:59:55.397100 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.397081    4314 scope.go:122] "RemoveContainer" containerID="3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"
Nov 01 08:59:55.398575 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.398526    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:55.398632 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.398616    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:55.398667 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:55.398650    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 08:59:56.047324 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.047289    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"]
Nov 01 08:59:56.050951 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.050930    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da49e37e-p5f8r"]
Nov 01 08:59:56.405771 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.405683    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" event={"ID":"0673bae6-a048-4b51-8444-988db5035dec","Type":"ContainerStarted","Data":"6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59"}
Nov 01 08:59:56.478554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.477951    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 08:59:56.478554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.478092    4314 patch_prober.go:29] interesting pod/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 08:59:56.478554 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.478133    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 08:59:56.933748 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:56.933719    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d61ece-5dbf-4161-958f-6ac825bf40df" path="/var/lib/kubelet/pods/27d61ece-5dbf-4161-958f-6ac825bf40df/volumes"
Nov 01 08:59:57.482665 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:57.482624    4314 patch_prober.go:29] interesting pod/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 08:59:57.483133 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:57.482688    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 08:59:58.383329 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:58.383295    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:58.487580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:58.484893    4314 patch_prober.go:29] interesting pod/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 08:59:58.487580 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:58.484973    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 08:59:59.057478 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:59.057445    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"]
Nov 01 08:59:59.103134 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:59.103100    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06533b1-7tm97"]
Nov 01 08:59:59.112407 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:59.112377    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 08:59:59.474995 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:59.474909    4314 patch_prober.go:29] interesting pod/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 08:59:59.475264 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:59.475235    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 08:59:59.981578 ip-10-0-134-93 kubenswrapper[4314]: I1101 08:59:59.981506    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:00.184968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.184924    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"]
Nov 01 09:00:00.185820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.185802    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9109b660-10be-47f2-a6f6-d39ebc028aed" containerName="revoke-keys"
Nov 01 09:00:00.185879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.185822    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9109b660-10be-47f2-a6f6-d39ebc028aed" containerName="revoke-keys"
Nov 01 09:00:00.185955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.185945    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9109b660-10be-47f2-a6f6-d39ebc028aed" containerName="revoke-keys"
Nov 01 09:00:00.189545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.189510    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.190398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.190376    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"]
Nov 01 09:00:00.194511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.194495    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.195770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.195751    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"]
Nov 01 09:00:00.200495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.200475    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.206289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.206257    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"]
Nov 01 09:00:00.210439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.210422    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"]
Nov 01 09:00:00.210591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.210572    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.214443 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.214426    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.225798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.225777    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"]
Nov 01 09:00:00.230608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.230592    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"]
Nov 01 09:00:00.230715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.230702    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.234651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.234632    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.238411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.238393    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:00:00.248412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.248396    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:00:00.248497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.248428    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:00:00.250290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250261    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.250414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250397    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btzp5\" (UniqueName: \"kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5\") pod \"maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.250480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250436    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.250580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250561    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vxb\" (UniqueName: \"kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.250642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250623    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.250712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250701    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.250747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250721    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djthv\" (UniqueName: \"kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.250747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250739    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmb2v\" (UniqueName: \"kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.250835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250760    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxvh\" (UniqueName: \"kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.250943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250926    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h652\" (UniqueName: \"kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652\") pod \"maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.251003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250957    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.251048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.250998    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.256706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.256685    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"]
Nov 01 09:00:00.284834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.284744    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"]
Nov 01 09:00:00.290083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.290059    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"]
Nov 01 09:00:00.312892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.307286    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"]
Nov 01 09:00:00.313038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.312962    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"]
Nov 01 09:00:00.354206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.353118    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vxb\" (UniqueName: \"kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.354206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.353595    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8trb\" (UniqueName: \"kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb\") pod \"maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.354206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.353739    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.354718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.354694    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.354828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.354745    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"]
Nov 01 09:00:00.354828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.353770    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.354945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.354899    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.354945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.354930    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djthv\" (UniqueName: \"kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.355045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.354960    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmb2v\" (UniqueName: \"kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.355045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.354994    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxvh\" (UniqueName: \"kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.355148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.355064    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h652\" (UniqueName: \"kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652\") pod \"maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.355148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.355091    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.355148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.355134    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.355301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.355217    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.355360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.355330    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btzp5\" (UniqueName: \"kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5\") pod \"maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.355411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.355367    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.356752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.356728    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.370554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.359969    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"]
Nov 01 09:00:00.384562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.384415    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.388556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.385059    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.390170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.390145    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.390256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.390192    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.415320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.415294    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vxb\" (UniqueName: \"kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.420400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.420377    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmb2v\" (UniqueName: \"kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.423952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.423305    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djthv\" (UniqueName: \"kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.423952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.423907    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxvh\" (UniqueName: \"kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.430204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.430176    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btzp5\" (UniqueName: \"kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5\") pod \"maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.431470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.431437    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h652\" (UniqueName: \"kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652\") pod \"maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.457121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.457095    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8trb\" (UniqueName: \"kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb\") pod \"maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.457275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.457161    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.482587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.457932    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.483584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.483114    4314 patch_prober.go:29] interesting pod/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 09:00:00.483584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.483169    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 09:00:00.483584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.483405    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:00.507314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.507295    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:00.514748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.514725    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8trb\" (UniqueName: \"kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb\") pod \"maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.516571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.516418    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:00.523453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.523434    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:00.538632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.538605    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:00.538998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.538959    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:00.548448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.548425    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:00.554614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.554576    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:00.657102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.657072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:00.947799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:00.946434    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182b8531-8a90-4d44-ab97-3fa8239d300b" path="/var/lib/kubelet/pods/182b8531-8a90-4d44-ab97-3fa8239d300b/volumes"
Nov 01 09:00:01.228862 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.228825    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaf3c37_a1a0_4b86_9e45_b4b7f2a78225.slice/crio-8807a8fe8430d0991dd289575504f35efd2cd2d3085f1ce1de32439bd48a9a56 WatchSource:0}: Error finding container 8807a8fe8430d0991dd289575504f35efd2cd2d3085f1ce1de32439bd48a9a56: Status 404 returned error can't find the container with id 8807a8fe8430d0991dd289575504f35efd2cd2d3085f1ce1de32439bd48a9a56
Nov 01 09:00:01.229191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.229117    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"]
Nov 01 09:00:01.273977 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.273950    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6f5f91_c9d1_404b_bc75_6dbd7d03e7f7.slice/crio-db318deebbc7f40054e94d1d47a53f59ae4b15758b35ba11ee4077563e510240 WatchSource:0}: Error finding container db318deebbc7f40054e94d1d47a53f59ae4b15758b35ba11ee4077563e510240: Status 404 returned error can't find the container with id db318deebbc7f40054e94d1d47a53f59ae4b15758b35ba11ee4077563e510240
Nov 01 09:00:01.275491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.275468    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"]
Nov 01 09:00:01.280646 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.280623    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eb33b6_4f4e_4823_b4d8_1506fea525c1.slice/crio-fb7184aa6d9f96c4007fc2fc03ae3feb4d6c9db655ac569c642db082abe21e71 WatchSource:0}: Error finding container fb7184aa6d9f96c4007fc2fc03ae3feb4d6c9db655ac569c642db082abe21e71: Status 404 returned error can't find the container with id fb7184aa6d9f96c4007fc2fc03ae3feb4d6c9db655ac569c642db082abe21e71
Nov 01 09:00:01.281188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.281164    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"]
Nov 01 09:00:01.281692 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.281669    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44466dc_a198_4995_86fb_2fc47266a514.slice/crio-9edc6f644dd1072383b2a658feb2fd98c0751bb5a4d26f9beb2d5aab6acf93b3 WatchSource:0}: Error finding container 9edc6f644dd1072383b2a658feb2fd98c0751bb5a4d26f9beb2d5aab6acf93b3: Status 404 returned error can't find the container with id 9edc6f644dd1072383b2a658feb2fd98c0751bb5a4d26f9beb2d5aab6acf93b3
Nov 01 09:00:01.287743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.287718    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"]
Nov 01 09:00:01.324823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.324797    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"]
Nov 01 09:00:01.327777 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.327755    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780e6ce7_e406_4e5d_af0b_875e24a32bc4.slice/crio-8523eb75c1e057c96b0bec7ee4578e13f4111c6bb9d67159bc1bca5084f7e1eb WatchSource:0}: Error finding container 8523eb75c1e057c96b0bec7ee4578e13f4111c6bb9d67159bc1bca5084f7e1eb: Status 404 returned error can't find the container with id 8523eb75c1e057c96b0bec7ee4578e13f4111c6bb9d67159bc1bca5084f7e1eb
Nov 01 09:00:01.366955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.366917    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"]
Nov 01 09:00:01.452078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.452048    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"]
Nov 01 09:00:01.463651 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.463624    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e47389_7265_4de0_9cef_78d6bd2c1399.slice/crio-8bf4723d59076aadbf6de6667a543f39e606c40c1cf1147c1910577c0301465b WatchSource:0}: Error finding container 8bf4723d59076aadbf6de6667a543f39e606c40c1cf1147c1910577c0301465b: Status 404 returned error can't find the container with id 8bf4723d59076aadbf6de6667a543f39e606c40c1cf1147c1910577c0301465b
Nov 01 09:00:01.464055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.464025    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerStarted","Data":"9edc6f644dd1072383b2a658feb2fd98c0751bb5a4d26f9beb2d5aab6acf93b3"}
Nov 01 09:00:01.466698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.466676    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerStarted","Data":"8807a8fe8430d0991dd289575504f35efd2cd2d3085f1ce1de32439bd48a9a56"}
Nov 01 09:00:01.467755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.467737    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerStarted","Data":"db318deebbc7f40054e94d1d47a53f59ae4b15758b35ba11ee4077563e510240"}
Nov 01 09:00:01.469229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.469210    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerStarted","Data":"8523eb75c1e057c96b0bec7ee4578e13f4111c6bb9d67159bc1bca5084f7e1eb"}
Nov 01 09:00:01.470290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.470272    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerStarted","Data":"fb7184aa6d9f96c4007fc2fc03ae3feb4d6c9db655ac569c642db082abe21e71"}
Nov 01 09:00:01.499159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.499128    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 09:00:01.499487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.499463    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 09:00:01.501215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:01.501193    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 09:00:01.560231 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:01.560012    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9841d8_5f7e_4743_b18d_71f37efd9ab0.slice/crio-17b14aa381e568cf8b9d1cef427293a45f2cfc711b1f656c6bc140aebde31dec WatchSource:0}: Error finding container 17b14aa381e568cf8b9d1cef427293a45f2cfc711b1f656c6bc140aebde31dec: Status 404 returned error can't find the container with id 17b14aa381e568cf8b9d1cef427293a45f2cfc711b1f656c6bc140aebde31dec
Nov 01 09:00:02.480005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.479972    4314 generic.go:356] "Generic (PLEG): container finished" podID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerID="1a3d0122266080cbb4f1ed3c35964e5e5559ff7b420a1c822d82c46984b7959f" exitCode=60
Nov 01 09:00:02.480390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.480083    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerDied","Data":"1a3d0122266080cbb4f1ed3c35964e5e5559ff7b420a1c822d82c46984b7959f"}
Nov 01 09:00:02.480557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.480510    4314 scope.go:122] "RemoveContainer" containerID="1a3d0122266080cbb4f1ed3c35964e5e5559ff7b420a1c822d82c46984b7959f"
Nov 01 09:00:02.481972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.481947    4314 generic.go:356] "Generic (PLEG): container finished" podID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerID="9bb9bf76ba9065208ac38dd98ccc7372594186536586de6d8a97d994ff48680e" exitCode=60
Nov 01 09:00:02.482077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.482032    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerDied","Data":"9bb9bf76ba9065208ac38dd98ccc7372594186536586de6d8a97d994ff48680e"}
Nov 01 09:00:02.482316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.482295    4314 scope.go:122] "RemoveContainer" containerID="9bb9bf76ba9065208ac38dd98ccc7372594186536586de6d8a97d994ff48680e"
Nov 01 09:00:02.483755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.483722    4314 generic.go:356] "Generic (PLEG): container finished" podID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerID="661048f8a192e97143d7efbf53e461a6634a2d9ad5729d803592e11b0b6fdbd5" exitCode=60
Nov 01 09:00:02.483937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.483808    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerDied","Data":"661048f8a192e97143d7efbf53e461a6634a2d9ad5729d803592e11b0b6fdbd5"}
Nov 01 09:00:02.484077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.484062    4314 scope.go:122] "RemoveContainer" containerID="661048f8a192e97143d7efbf53e461a6634a2d9ad5729d803592e11b0b6fdbd5"
Nov 01 09:00:02.485928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.485837    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerStarted","Data":"453d0bd936a065282b77c70a96815e89cb0161174c3fb0c08f11621227987566"}
Nov 01 09:00:02.485928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.485873    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerStarted","Data":"17b14aa381e568cf8b9d1cef427293a45f2cfc711b1f656c6bc140aebde31dec"}
Nov 01 09:00:02.487707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.487680    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerStarted","Data":"9cf9ee59ef2b04614574dab6927a25e74f4cbb8f8fa0ee98b64d46b2e72c38b7"}
Nov 01 09:00:02.487806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.487730    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerStarted","Data":"8bf4723d59076aadbf6de6667a543f39e606c40c1cf1147c1910577c0301465b"}
Nov 01 09:00:02.489979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.489959    4314 generic.go:356] "Generic (PLEG): container finished" podID="c44466dc-a198-4995-86fb-2fc47266a514" containerID="185bfbc0b0a17ac3f8c45aa274f485a845c79156be7af8c6ee1efeca8b406db9" exitCode=60
Nov 01 09:00:02.490073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.490034    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerDied","Data":"185bfbc0b0a17ac3f8c45aa274f485a845c79156be7af8c6ee1efeca8b406db9"}
Nov 01 09:00:02.490226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.490209    4314 scope.go:122] "RemoveContainer" containerID="185bfbc0b0a17ac3f8c45aa274f485a845c79156be7af8c6ee1efeca8b406db9"
Nov 01 09:00:02.492390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.492368    4314 generic.go:356] "Generic (PLEG): container finished" podID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerID="d1e2f51e5d808c95e08e55b77191df756202715e87638191f039516e0642a931" exitCode=60
Nov 01 09:00:02.492483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.492466    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerDied","Data":"d1e2f51e5d808c95e08e55b77191df756202715e87638191f039516e0642a931"}
Nov 01 09:00:02.492762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.492741    4314 scope.go:122] "RemoveContainer" containerID="d1e2f51e5d808c95e08e55b77191df756202715e87638191f039516e0642a931"
Nov 01 09:00:02.812842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.812723    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" podStartSLOduration=2.81270139 podStartE2EDuration="2.81270139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:02.802934749 +0000 UTC m=+1675.997329303" watchObservedRunningTime="2026-11-01 09:00:02.81270139 +0000 UTC m=+1676.007095946"
Nov 01 09:00:02.918262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:02.918212    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" podStartSLOduration=2.918196507 podStartE2EDuration="2.918196507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:02.913284962 +0000 UTC m=+1676.107679521" watchObservedRunningTime="2026-11-01 09:00:02.918196507 +0000 UTC m=+1676.112591095"
Nov 01 09:00:03.502609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.502573    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerStarted","Data":"318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d"}
Nov 01 09:00:03.504597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.504564    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerStarted","Data":"58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764"}
Nov 01 09:00:03.506596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.506574    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerStarted","Data":"77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad"}
Nov 01 09:00:03.508341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.508297    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerStarted","Data":"5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b"}
Nov 01 09:00:03.510040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.510020    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerStarted","Data":"b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea"}
Nov 01 09:00:03.511504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.511482    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerID="453d0bd936a065282b77c70a96815e89cb0161174c3fb0c08f11621227987566" exitCode=60
Nov 01 09:00:03.511629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.511563    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerDied","Data":"453d0bd936a065282b77c70a96815e89cb0161174c3fb0c08f11621227987566"}
Nov 01 09:00:03.511754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.511737    4314 scope.go:122] "RemoveContainer" containerID="453d0bd936a065282b77c70a96815e89cb0161174c3fb0c08f11621227987566"
Nov 01 09:00:03.513083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.513060    4314 generic.go:356] "Generic (PLEG): container finished" podID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerID="9cf9ee59ef2b04614574dab6927a25e74f4cbb8f8fa0ee98b64d46b2e72c38b7" exitCode=60
Nov 01 09:00:03.513188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.513087    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerDied","Data":"9cf9ee59ef2b04614574dab6927a25e74f4cbb8f8fa0ee98b64d46b2e72c38b7"}
Nov 01 09:00:03.513362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.513318    4314 scope.go:122] "RemoveContainer" containerID="9cf9ee59ef2b04614574dab6927a25e74f4cbb8f8fa0ee98b64d46b2e72c38b7"
Nov 01 09:00:03.611852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.611803    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" podStartSLOduration=3.611789384 podStartE2EDuration="3.611789384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:03.609786291 +0000 UTC m=+1676.804180844" watchObservedRunningTime="2026-11-01 09:00:03.611789384 +0000 UTC m=+1676.806183937"
Nov 01 09:00:03.940195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:03.940140    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" podStartSLOduration=3.940122282 podStartE2EDuration="3.940122282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:03.93914538 +0000 UTC m=+1677.133539935" watchObservedRunningTime="2026-11-01 09:00:03.940122282 +0000 UTC m=+1677.134516837"
Nov 01 09:00:04.050569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.050473    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" podStartSLOduration=4.050459252 podStartE2EDuration="4.050459252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:04.046167666 +0000 UTC m=+1677.240562232" watchObservedRunningTime="2026-11-01 09:00:04.050459252 +0000 UTC m=+1677.244853827"
Nov 01 09:00:04.156444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.156392    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" podStartSLOduration=4.15637741 podStartE2EDuration="4.15637741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:04.153928197 +0000 UTC m=+1677.348322750" watchObservedRunningTime="2026-11-01 09:00:04.15637741 +0000 UTC m=+1677.350772008"
Nov 01 09:00:04.266147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.266089    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" podStartSLOduration=4.26607229 podStartE2EDuration="4.26607229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:04.264688915 +0000 UTC m=+1677.459083470" watchObservedRunningTime="2026-11-01 09:00:04.26607229 +0000 UTC m=+1677.460466844"
Nov 01 09:00:04.297001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.296927    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:04.527868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.527833    4314 generic.go:356] "Generic (PLEG): container finished" podID="c44466dc-a198-4995-86fb-2fc47266a514" containerID="318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d" exitCode=60
Nov 01 09:00:04.528364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.527923    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerDied","Data":"318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d"}
Nov 01 09:00:04.528364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.527992    4314 scope.go:122] "RemoveContainer" containerID="185bfbc0b0a17ac3f8c45aa274f485a845c79156be7af8c6ee1efeca8b406db9"
Nov 01 09:00:04.528364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.528260    4314 scope.go:122] "RemoveContainer" containerID="318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d"
Nov 01 09:00:04.528760 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:04.528732    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5_odh-ai-gateway-infra(c44466dc-a198-4995-86fb-2fc47266a514)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" podUID="c44466dc-a198-4995-86fb-2fc47266a514"
Nov 01 09:00:04.530358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.530337    4314 generic.go:356] "Generic (PLEG): container finished" podID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerID="58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764" exitCode=60
Nov 01 09:00:04.530472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.530421    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerDied","Data":"58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764"}
Nov 01 09:00:04.530701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.530686    4314 scope.go:122] "RemoveContainer" containerID="58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764"
Nov 01 09:00:04.531083 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:04.531060    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x_odh-ai-gateway-infra(bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225"
Nov 01 09:00:04.533476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.532682    4314 generic.go:356] "Generic (PLEG): container finished" podID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerID="77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad" exitCode=60
Nov 01 09:00:04.533476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.532765    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerDied","Data":"77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad"}
Nov 01 09:00:04.533476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.533101    4314 scope.go:122] "RemoveContainer" containerID="77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad"
Nov 01 09:00:04.533476 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:04.533407    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr_odh-ai-gateway-infra(ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7"
Nov 01 09:00:04.536659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.536625    4314 generic.go:356] "Generic (PLEG): container finished" podID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerID="5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b" exitCode=60
Nov 01 09:00:04.536816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.536666    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerDied","Data":"5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b"}
Nov 01 09:00:04.536993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.536970    4314 scope.go:122] "RemoveContainer" containerID="5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b"
Nov 01 09:00:04.537332 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:04.537280    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn_odh-ai-gateway-infra(780e6ce7-e406-4e5d-af0b-875e24a32bc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4"
Nov 01 09:00:04.538907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.538888    4314 generic.go:356] "Generic (PLEG): container finished" podID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerID="b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea" exitCode=60
Nov 01 09:00:04.539006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.538956    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerDied","Data":"b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea"}
Nov 01 09:00:04.539167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.539154    4314 scope.go:122] "RemoveContainer" containerID="b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea"
Nov 01 09:00:04.539436 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:04.539415    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v_odh-ai-gateway-infra(19eb33b6-4f4e-4823-b4d8-1506fea525c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1"
Nov 01 09:00:04.540949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.540910    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerStarted","Data":"f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa"}
Nov 01 09:00:04.542983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.542965    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerStarted","Data":"af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78"}
Nov 01 09:00:04.543948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.543932    4314 scope.go:122] "RemoveContainer" containerID="d1e2f51e5d808c95e08e55b77191df756202715e87638191f039516e0642a931"
Nov 01 09:00:04.561744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.559082    4314 scope.go:122] "RemoveContainer" containerID="1a3d0122266080cbb4f1ed3c35964e5e5559ff7b420a1c822d82c46984b7959f"
Nov 01 09:00:04.701193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.701162    4314 scope.go:122] "RemoveContainer" containerID="9bb9bf76ba9065208ac38dd98ccc7372594186536586de6d8a97d994ff48680e"
Nov 01 09:00:04.728385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:04.728191    4314 scope.go:122] "RemoveContainer" containerID="661048f8a192e97143d7efbf53e461a6634a2d9ad5729d803592e11b0b6fdbd5"
Nov 01 09:00:05.457691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.457660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:05.554586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.554554    4314 scope.go:122] "RemoveContainer" containerID="77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad"
Nov 01 09:00:05.555032 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.554983    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr_odh-ai-gateway-infra(ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7"
Nov 01 09:00:05.556446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.556421    4314 scope.go:122] "RemoveContainer" containerID="5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b"
Nov 01 09:00:05.556815 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.556788    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn_odh-ai-gateway-infra(780e6ce7-e406-4e5d-af0b-875e24a32bc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4"
Nov 01 09:00:05.558083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.558061    4314 scope.go:122] "RemoveContainer" containerID="b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea"
Nov 01 09:00:05.558395 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.558376    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v_odh-ai-gateway-infra(19eb33b6-4f4e-4823-b4d8-1506fea525c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1"
Nov 01 09:00:05.559649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.559629    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerID="f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa" exitCode=60
Nov 01 09:00:05.559761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.559709    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerDied","Data":"f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa"}
Nov 01 09:00:05.559834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.559807    4314 scope.go:122] "RemoveContainer" containerID="453d0bd936a065282b77c70a96815e89cb0161174c3fb0c08f11621227987566"
Nov 01 09:00:05.559962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.559946    4314 scope.go:122] "RemoveContainer" containerID="f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa"
Nov 01 09:00:05.560226 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.560205    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x_odh-ai-gateway-infra(1f9841d8-5f7e-4743-b18d-71f37efd9ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0"
Nov 01 09:00:05.561780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.561761    4314 generic.go:356] "Generic (PLEG): container finished" podID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerID="af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78" exitCode=60
Nov 01 09:00:05.561879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.561837    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerDied","Data":"af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78"}
Nov 01 09:00:05.562093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.562077    4314 scope.go:122] "RemoveContainer" containerID="af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78"
Nov 01 09:00:05.562427 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.562404    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f_odh-ai-gateway-infra(e5e47389-7265-4de0-9cef-78d6bd2c1399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399"
Nov 01 09:00:05.563764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.563747    4314 scope.go:122] "RemoveContainer" containerID="318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d"
Nov 01 09:00:05.564045 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.564026    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5_odh-ai-gateway-infra(c44466dc-a198-4995-86fb-2fc47266a514)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" podUID="c44466dc-a198-4995-86fb-2fc47266a514"
Nov 01 09:00:05.565397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.565381    4314 scope.go:122] "RemoveContainer" containerID="58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764"
Nov 01 09:00:05.565661 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:05.565637    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x_odh-ai-gateway-infra(bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225"
Nov 01 09:00:05.574462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.574445    4314 scope.go:122] "RemoveContainer" containerID="9cf9ee59ef2b04614574dab6927a25e74f4cbb8f8fa0ee98b64d46b2e72c38b7"
Nov 01 09:00:05.616521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.616495    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:05.804554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:05.802949    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:06.584142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:06.584117    4314 scope.go:122] "RemoveContainer" containerID="f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa"
Nov 01 09:00:06.585023 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:06.584998    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x_odh-ai-gateway-infra(1f9841d8-5f7e-4743-b18d-71f37efd9ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0"
Nov 01 09:00:06.588040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:06.588023    4314 scope.go:122] "RemoveContainer" containerID="af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78"
Nov 01 09:00:06.588438 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:06.588417    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f_odh-ai-gateway-infra(e5e47389-7265-4de0-9cef-78d6bd2c1399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399"
Nov 01 09:00:07.050777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.050745    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"]
Nov 01 09:00:07.050996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.050971    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" containerID="cri-o://e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842" gracePeriod=30
Nov 01 09:00:07.060218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.060164    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"]
Nov 01 09:00:07.060393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.060374    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" containerID="cri-o://bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad" gracePeriod=30
Nov 01 09:00:07.064386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.064337    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"]
Nov 01 09:00:07.064611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.064589    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy" containerID="cri-o://bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9" gracePeriod=30
Nov 01 09:00:07.068392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.068371    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"]
Nov 01 09:00:07.068578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.068558    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" containerID="cri-o://23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db" gracePeriod=30
Nov 01 09:00:07.535121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.535087    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:07.569700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.569673    4314 scope.go:122] "RemoveContainer" containerID="6eea4118fe453fda435ae7cdeed0ffde81cfca8a2cefcf640f46b99aaa70b5f6"
Nov 01 09:00:07.582983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.582954    4314 scope.go:122] "RemoveContainer" containerID="b3aab14e6ea4f338e40a6e50b4cee8d591db7f762fa68a615abb2e5cdf540213"
Nov 01 09:00:07.591974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.591954    4314 scope.go:122] "RemoveContainer" containerID="d8b980ce40672354ca5a3223a1ca50c1f3a4aa999e6a09790b12e3467172aedc"
Nov 01 09:00:07.603371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.603347    4314 scope.go:122] "RemoveContainer" containerID="c8eb1ee3b9513c2a194469a4d8fd479473f8dc74e69ae214228e90cad6e89b75"
Nov 01 09:00:07.611733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.611711    4314 scope.go:122] "RemoveContainer" containerID="e0a7295743b56fc35eac9826ecabd1dccc7536f6c4268f3d63018ac9c8970228"
Nov 01 09:00:07.620732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.620712    4314 scope.go:122] "RemoveContainer" containerID="c952349ba813f8b3161bba42313ba4279be10cfc36fedcecb615bc6b5332280a"
Nov 01 09:00:07.641090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.641063    4314 scope.go:122] "RemoveContainer" containerID="c14486d788db31a9384f3d0306aa31e743cadb586540ed18ff46ad069fc646c9"
Nov 01 09:00:07.658235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.658207    4314 scope.go:122] "RemoveContainer" containerID="3c7322f55113663329370ea216cf9ebca3413113126c3cfbc742c8b18769fdde"
Nov 01 09:00:07.941767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:07.941694    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:08.058569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:08.058148    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"]
Nov 01 09:00:08.065058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:08.065025    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e41c0-vdstr"]
Nov 01 09:00:08.096246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:08.096214    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:08.619239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:08.619204    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:08.811638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:08.799981    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:08.933879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:08.933790    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf117cc-8ab5-47f7-ab22-1a76da47a1bc" path="/var/lib/kubelet/pods/baf117cc-8ab5-47f7-ab22-1a76da47a1bc/volumes"
Nov 01 09:00:09.432506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:09.432457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:09.991660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:09.991444    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:00:09.991660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:09.991498    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:00:10.104633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:10.104596    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:10.611979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:10.611948    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"]
Nov 01 09:00:10.612198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:10.612177    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy" containerID="cri-o://059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55" gracePeriod=30
Nov 01 09:00:10.617368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:10.617341    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:11.325008 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:11.324976    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:00:12.073060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.073029    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 09:00:12.073217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.073104    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 09:00:12.090255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.090222    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": read tcp 10.132.0.2:59334->10.132.0.84:15021: read: connection reset by peer" start-of-body=
Nov 01 09:00:12.090368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.090286    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": read tcp 10.132.0.2:59334->10.132.0.84:15021: read: connection reset by peer"
Nov 01 09:00:12.350793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.350624    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 09:00:12.390618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.390593    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 09:00:12.453037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453008    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.453181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453056    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.453181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453084    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.453181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453131    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.453181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453160    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.453490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453370    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.453626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453594    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.453739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453646    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.453739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453663    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.453739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453675    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.453739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453735    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.454007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453822    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.454007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453883    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.454007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453889    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.454007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453941    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.454007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.453998    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.454322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454040    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.454322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454103    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.454322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454154    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.454322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454232    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.454322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454291    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy\") pod \"e44ff955-8bb9-481d-94d1-377fdb20961d\" (UID: \"e44ff955-8bb9-481d-94d1-377fdb20961d\") "
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454345    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454410    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token\") pod \"35061eb7-9af6-46fd-a0f1-f3358f762de6\" (UID: \"35061eb7-9af6-46fd-a0f1-f3358f762de6\") "
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454703    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454789    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.454909    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455101    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455247    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455274    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455288    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455301    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455313    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455325    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455338    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455354    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455367    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44ff955-8bb9-481d-94d1-377fdb20961d-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.455641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455643    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:12.456524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455769    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "kube-api-access-4kkqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.456524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.455993    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "kube-api-access-cdh77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.457004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.456979    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:12.457115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.457002    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:12.457196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.457173    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.457584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.457514    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.458092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.458069    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token" pod "35061eb7-9af6-46fd-a0f1-f3358f762de6" (UID: "35061eb7-9af6-46fd-a0f1-f3358f762de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.458291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.458275    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token" pod "e44ff955-8bb9-481d-94d1-377fdb20961d" (UID: "e44ff955-8bb9-481d-94d1-377fdb20961d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.465054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.465037    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 09:00:12.467469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.467454    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556194    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556243    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556346    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556401    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556443    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556481    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556517    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556595    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.556798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.556645    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.558661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.558358    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:12.559097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.558888    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.559202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559125    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.559291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559161    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:12.559291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559218    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.559461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559433    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.559863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559487    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.559863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559642    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.559863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559673    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.559863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559737    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.559863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559859    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.560262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559896    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.560262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.559928    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.560262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.560060    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.560262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.560166    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.560262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.560222    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.560625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.560297    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo\") pod \"e6432463-4ec0-4323-98e8-4ea25e21c415\" (UID: \"e6432463-4ec0-4323-98e8-4ea25e21c415\") "
Nov 01 09:00:12.560625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.560385    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns\") pod \"d694432c-21e5-4c92-be3f-d4674d18b15b\" (UID: \"d694432c-21e5-4c92-be3f-d4674d18b15b\") "
Nov 01 09:00:12.560844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.560820    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.561069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561049    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.561911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561886    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561918    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561936    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561950    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561964    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561979    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdh77\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-kube-api-access-cdh77\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.561994    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kkqx\" (UniqueName: \"kubernetes.io/projected/35061eb7-9af6-46fd-a0f1-f3358f762de6-kube-api-access-4kkqx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562007    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562020    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562035    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562048    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562063    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35061eb7-9af6-46fd-a0f1-f3358f762de6-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562076    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35061eb7-9af6-46fd-a0f1-f3358f762de6-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562089    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562103    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d694432c-21e5-4c92-be3f-d4674d18b15b-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562117    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562130    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562143    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6432463-4ec0-4323-98e8-4ea25e21c415-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562157    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44ff955-8bb9-481d-94d1-377fdb20961d-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.562755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.562170    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.564168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.564136    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:12.564923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.564899    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.565090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.565065    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:12.565242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.565171    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "kube-api-access-qrgns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.565684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.565661    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "kube-api-access-g8p7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.566194 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.566171    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token" pod "e6432463-4ec0-4323-98e8-4ea25e21c415" (UID: "e6432463-4ec0-4323-98e8-4ea25e21c415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.566308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.566290    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token" pod "d694432c-21e5-4c92-be3f-d4674d18b15b" (UID: "d694432c-21e5-4c92-be3f-d4674d18b15b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.651091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.651009    4314 generic.go:356] "Generic (PLEG): container finished" podID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerID="e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842" exitCode=0
Nov 01 09:00:12.651091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.651071    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"
Nov 01 09:00:12.651320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.651093    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" event={"ID":"35061eb7-9af6-46fd-a0f1-f3358f762de6","Type":"ContainerDied","Data":"e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842"}
Nov 01 09:00:12.651320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.651158    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv" event={"ID":"35061eb7-9af6-46fd-a0f1-f3358f762de6","Type":"ContainerDied","Data":"5c4885d574c0fe02fb8e307d205e8e2ad0f8b0745c170a853e79483caab690c7"}
Nov 01 09:00:12.651320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.651188    4314 scope.go:122] "RemoveContainer" containerID="e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842"
Nov 01 09:00:12.652911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.652735    4314 generic.go:356] "Generic (PLEG): container finished" podID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerID="23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db" exitCode=0
Nov 01 09:00:12.652911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.652843    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" event={"ID":"d694432c-21e5-4c92-be3f-d4674d18b15b","Type":"ContainerDied","Data":"23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db"}
Nov 01 09:00:12.652911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.652884    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6" event={"ID":"d694432c-21e5-4c92-be3f-d4674d18b15b","Type":"ContainerDied","Data":"355f84a75dc7a49ba7e360ae9a6e748f392bdfd6d1ecae7aee666bb7a0093dfc"}
Nov 01 09:00:12.653174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.652959    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"
Nov 01 09:00:12.661581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.656598    4314 generic.go:356] "Generic (PLEG): container finished" podID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerID="bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad" exitCode=0
Nov 01 09:00:12.661581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.656683    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" event={"ID":"e6432463-4ec0-4323-98e8-4ea25e21c415","Type":"ContainerDied","Data":"bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad"}
Nov 01 09:00:12.661581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.656734    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp" event={"ID":"e6432463-4ec0-4323-98e8-4ea25e21c415","Type":"ContainerDied","Data":"8af268935714bbd3e98f7d2367dfca9f72d185d4bf198b6676aba372231f5623"}
Nov 01 09:00:12.661581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.656808    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665372    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665395    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8p7v\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-kube-api-access-g8p7v\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665409    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665422    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665435    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6432463-4ec0-4323-98e8-4ea25e21c415-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665447    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgns\" (UniqueName: \"kubernetes.io/projected/d694432c-21e5-4c92-be3f-d4674d18b15b-kube-api-access-qrgns\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.666672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.665459    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d694432c-21e5-4c92-be3f-d4674d18b15b-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.668340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.668257    4314 generic.go:356] "Generic (PLEG): container finished" podID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerID="bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9" exitCode=0
Nov 01 09:00:12.668413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.668350    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" event={"ID":"e44ff955-8bb9-481d-94d1-377fdb20961d","Type":"ContainerDied","Data":"bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9"}
Nov 01 09:00:12.668413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.668394    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj" event={"ID":"e44ff955-8bb9-481d-94d1-377fdb20961d","Type":"ContainerDied","Data":"601a9dc7a604529e13749839ebc1259820280f17d52c516d93eae046912b0f29"}
Nov 01 09:00:12.669594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.668462    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"
Nov 01 09:00:12.670734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.670699    4314 scope.go:122] "RemoveContainer" containerID="e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842"
Nov 01 09:00:12.673685 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:12.673655    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842\": container with ID starting with e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842 not found: ID does not exist" containerID="e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842"
Nov 01 09:00:12.673779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.673695    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842"} err="failed to get container status \"e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842\": rpc error: code = NotFound desc = could not find container \"e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842\": container with ID starting with e42ba9882d89ed9471b50f0f469f12f84618a31848514023bdcfd01f186ea842 not found: ID does not exist"
Nov 01 09:00:12.673779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.673720    4314 scope.go:122] "RemoveContainer" containerID="23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db"
Nov 01 09:00:12.688782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.688711    4314 scope.go:122] "RemoveContainer" containerID="23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db"
Nov 01 09:00:12.689116 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:12.689093    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db\": container with ID starting with 23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db not found: ID does not exist" containerID="23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db"
Nov 01 09:00:12.689228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.689132    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db"} err="failed to get container status \"23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db\": rpc error: code = NotFound desc = could not find container \"23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db\": container with ID starting with 23c95b35e2029fb1f6f1981c09e902032b262466c23c62a148741447974bf7db not found: ID does not exist"
Nov 01 09:00:12.689228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.689149    4314 scope.go:122] "RemoveContainer" containerID="bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad"
Nov 01 09:00:12.698646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.698627    4314 scope.go:122] "RemoveContainer" containerID="bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad"
Nov 01 09:00:12.698916 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:12.698898    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad\": container with ID starting with bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad not found: ID does not exist" containerID="bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad"
Nov 01 09:00:12.699013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.698919    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad"} err="failed to get container status \"bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad\": rpc error: code = NotFound desc = could not find container \"bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad\": container with ID starting with bf8b4fb8c041a0773f95ab75aa435248f9e21b0723e947d4980fe164ad9464ad not found: ID does not exist"
Nov 01 09:00:12.699013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.698938    4314 scope.go:122] "RemoveContainer" containerID="bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9"
Nov 01 09:00:12.708190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.708174    4314 scope.go:122] "RemoveContainer" containerID="bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9"
Nov 01 09:00:12.708480 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:12.708461    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9\": container with ID starting with bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9 not found: ID does not exist" containerID="bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9"
Nov 01 09:00:12.708559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.708489    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9"} err="failed to get container status \"bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9\": rpc error: code = NotFound desc = could not find container \"bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9\": container with ID starting with bae189f0b42c97f5fc66593462f130f02d84a1892a8c1fbf3fc4326d3ae522c9 not found: ID does not exist"
Nov 01 09:00:12.724959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.724939    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:12.876901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.876064    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:12.979697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.979610    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"]
Nov 01 09:00:12.984260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:12.984229    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-74tsv"]
Nov 01 09:00:13.079207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:13.079177    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"]
Nov 01 09:00:13.086262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:13.086240    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-r6svp"]
Nov 01 09:00:13.188607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:13.188572    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"]
Nov 01 09:00:13.192458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:13.192426    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-pthn6"]
Nov 01 09:00:13.293150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:13.293118    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"]
Nov 01 09:00:13.297545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:13.297513    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-k4ctj"]
Nov 01 09:00:14.403494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.403465    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:14.888793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.888741    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:14.936182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.936152    4314 scope.go:122] "RemoveContainer" containerID="77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad"
Nov 01 09:00:14.940368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.940019    4314 scope.go:122] "RemoveContainer" containerID="58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764"
Nov 01 09:00:14.941128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.940496    4314 scope.go:122] "RemoveContainer" containerID="f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa"
Nov 01 09:00:14.984556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.984489    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" path="/var/lib/kubelet/pods/35061eb7-9af6-46fd-a0f1-f3358f762de6/volumes"
Nov 01 09:00:14.985317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.985291    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" path="/var/lib/kubelet/pods/d694432c-21e5-4c92-be3f-d4674d18b15b/volumes"
Nov 01 09:00:14.986198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.986175    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" path="/var/lib/kubelet/pods/e44ff955-8bb9-481d-94d1-377fdb20961d/volumes"
Nov 01 09:00:14.987406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:14.987383    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" path="/var/lib/kubelet/pods/e6432463-4ec0-4323-98e8-4ea25e21c415/volumes"
Nov 01 09:00:15.645401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:15.645363    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:15.717355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:15.717259    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerStarted","Data":"361219e2ca3344c04a8496118ffe54a77d6c31ff130a40daa03e15587753538c"}
Nov 01 09:00:15.720228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:15.720171    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerStarted","Data":"af3aa4e0a22e56e8d24ab2a1e904f2d96c0de1b5a18587c6fb169b1a8b634235"}
Nov 01 09:00:15.722860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:15.722834    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerStarted","Data":"a9f8bc3cb749e96ec3705bd75fdb7ecaf58f78335ca3e038086dd9a0939666cd"}
Nov 01 09:00:15.921645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:15.921618    4314 scope.go:122] "RemoveContainer" containerID="b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea"
Nov 01 09:00:15.923350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:15.923334    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 09:00:16.022315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022294    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022354    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022429    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022453    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022491    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022561    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022607    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022668    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.022785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022686    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.022785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.022738    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket\") pod \"9ce33414-ccd3-4630-acc2-81c87015071a\" (UID: \"9ce33414-ccd3-4630-acc2-81c87015071a\") "
Nov 01 09:00:16.023053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.023029    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.023135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.023097    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:16.024499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.023235    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.024499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.024021    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.024711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.024610    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.024711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.024677    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.024848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.024795    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.024848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.024817    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce33414-ccd3-4630-acc2-81c87015071a-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.025010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.024990    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:16.025147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.025125    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.025260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.025237    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:16.025300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.025282    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq" pod "9ce33414-ccd3-4630-acc2-81c87015071a" (UID: "9ce33414-ccd3-4630-acc2-81c87015071a"). InnerVolumeSpecName "kube-api-access-rwftq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:16.126295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.126265    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwftq\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-kube-api-access-rwftq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.126295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.126294    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.126439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.126305    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce33414-ccd3-4630-acc2-81c87015071a-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.126439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.126313    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce33414-ccd3-4630-acc2-81c87015071a-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.126439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.126322    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce33414-ccd3-4630-acc2-81c87015071a-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.733339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.733301    4314 generic.go:356] "Generic (PLEG): container finished" podID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerID="361219e2ca3344c04a8496118ffe54a77d6c31ff130a40daa03e15587753538c" exitCode=60
Nov 01 09:00:16.733842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.733371    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerDied","Data":"361219e2ca3344c04a8496118ffe54a77d6c31ff130a40daa03e15587753538c"}
Nov 01 09:00:16.733842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.733429    4314 scope.go:122] "RemoveContainer" containerID="77c52fd5b4fb302b2dd966bb7752c10b0aea415e73513c5ab94079d78d0ec4ad"
Nov 01 09:00:16.733842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.733801    4314 scope.go:122] "RemoveContainer" containerID="361219e2ca3344c04a8496118ffe54a77d6c31ff130a40daa03e15587753538c"
Nov 01 09:00:16.734107 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:16.734087    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr_odh-ai-gateway-infra(ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7"
Nov 01 09:00:16.735520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.735496    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerStarted","Data":"a553e16ce9b094c30a0eea22b4fa03ec455d89a5a8565894b36a79a611acde3c"}
Nov 01 09:00:16.737346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.737320    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerID="af3aa4e0a22e56e8d24ab2a1e904f2d96c0de1b5a18587c6fb169b1a8b634235" exitCode=60
Nov 01 09:00:16.737444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.737388    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerDied","Data":"af3aa4e0a22e56e8d24ab2a1e904f2d96c0de1b5a18587c6fb169b1a8b634235"}
Nov 01 09:00:16.737712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.737698    4314 scope.go:122] "RemoveContainer" containerID="af3aa4e0a22e56e8d24ab2a1e904f2d96c0de1b5a18587c6fb169b1a8b634235"
Nov 01 09:00:16.737929 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:16.737903    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x_odh-ai-gateway-infra(1f9841d8-5f7e-4743-b18d-71f37efd9ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0"
Nov 01 09:00:16.739302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.739284    4314 generic.go:356] "Generic (PLEG): container finished" podID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerID="a9f8bc3cb749e96ec3705bd75fdb7ecaf58f78335ca3e038086dd9a0939666cd" exitCode=60
Nov 01 09:00:16.739383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.739352    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerDied","Data":"a9f8bc3cb749e96ec3705bd75fdb7ecaf58f78335ca3e038086dd9a0939666cd"}
Nov 01 09:00:16.739711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.739688    4314 scope.go:122] "RemoveContainer" containerID="a9f8bc3cb749e96ec3705bd75fdb7ecaf58f78335ca3e038086dd9a0939666cd"
Nov 01 09:00:16.739918 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:16.739895    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x_odh-ai-gateway-infra(bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225"
Nov 01 09:00:16.741294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.741274    4314 generic.go:356] "Generic (PLEG): container finished" podID="9ce33414-ccd3-4630-acc2-81c87015071a" containerID="059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55" exitCode=0
Nov 01 09:00:16.741372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.741297    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" event={"ID":"9ce33414-ccd3-4630-acc2-81c87015071a","Type":"ContainerDied","Data":"059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55"}
Nov 01 09:00:16.741372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.741336    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"
Nov 01 09:00:16.741372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.741364    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f" event={"ID":"9ce33414-ccd3-4630-acc2-81c87015071a","Type":"ContainerDied","Data":"7e10128ab1159b706040f2bd7ee8688eec9bbedf6bece0e24b982308a3d0ea60"}
Nov 01 09:00:16.749951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.749930    4314 scope.go:122] "RemoveContainer" containerID="f24a36586802ea9c89eb808e4bc89e2b30d4ede43b60b2878fdd89b334cc24fa"
Nov 01 09:00:16.763095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.763081    4314 scope.go:122] "RemoveContainer" containerID="58174dd0685c88d97410e4c7511ad8a49388e48557ec1e29bf7ae8d3493b9764"
Nov 01 09:00:16.835573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.833884    4314 scope.go:122] "RemoveContainer" containerID="059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55"
Nov 01 09:00:16.839579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.839556    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"]
Nov 01 09:00:16.846872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.846849    4314 scope.go:122] "RemoveContainer" containerID="e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"
Nov 01 09:00:16.860951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.860901    4314 scope.go:122] "RemoveContainer" containerID="059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55"
Nov 01 09:00:16.861881 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:16.861851    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55\": container with ID starting with 059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55 not found: ID does not exist" containerID="059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55"
Nov 01 09:00:16.861989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.861888    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55"} err="failed to get container status \"059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55\": rpc error: code = NotFound desc = could not find container \"059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55\": container with ID starting with 059507ec941358faf19c2b40666a1a96332d790b71551117615d136262630b55 not found: ID does not exist"
Nov 01 09:00:16.861989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.861914    4314 scope.go:122] "RemoveContainer" containerID="e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"
Nov 01 09:00:16.862192 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:16.862172    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd\": container with ID starting with e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd not found: ID does not exist" containerID="e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"
Nov 01 09:00:16.862251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.862199    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd"} err="failed to get container status \"e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd\": rpc error: code = NotFound desc = could not find container \"e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd\": container with ID starting with e3513640203188cd406d09434204145306d2e60308901d8aeb6baac07f55f2fd not found: ID does not exist"
Nov 01 09:00:16.946288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:16.946255    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"]
Nov 01 09:00:17.055414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.055335    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"]
Nov 01 09:00:17.184204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.184168    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"]
Nov 01 09:00:17.190223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.190196    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8701a0-openshift-default-5498d94f77-tc69f"]
Nov 01 09:00:17.753117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.753086    4314 generic.go:356] "Generic (PLEG): container finished" podID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerID="a553e16ce9b094c30a0eea22b4fa03ec455d89a5a8565894b36a79a611acde3c" exitCode=60
Nov 01 09:00:17.753574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.753114    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerDied","Data":"a553e16ce9b094c30a0eea22b4fa03ec455d89a5a8565894b36a79a611acde3c"}
Nov 01 09:00:17.753574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.753207    4314 scope.go:122] "RemoveContainer" containerID="b82299057e305a205d84b26daf3dcfd781b0281024ad4cca751abf9cdb5db7ea"
Nov 01 09:00:17.753720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.753599    4314 scope.go:122] "RemoveContainer" containerID="a553e16ce9b094c30a0eea22b4fa03ec455d89a5a8565894b36a79a611acde3c"
Nov 01 09:00:17.753942 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:17.753922    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v_odh-ai-gateway-infra(19eb33b6-4f4e-4823-b4d8-1506fea525c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1"
Nov 01 09:00:17.901931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.901898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:17.922328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.922302    4314 scope.go:122] "RemoveContainer" containerID="5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b"
Nov 01 09:00:17.923692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.923677    4314 scope.go:122] "RemoveContainer" containerID="af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78"
Nov 01 09:00:17.924303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.924282    4314 scope.go:122] "RemoveContainer" containerID="318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d"
Nov 01 09:00:17.931121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:17.931084    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:18.075618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.075520    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:18.076670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.076644    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb\" (UniqueName: \"kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb\") pod \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") "
Nov 01 09:00:18.076914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.076890    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca\") pod \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\" (UID: \"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7\") "
Nov 01 09:00:18.078435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.078407    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca" pod "ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" (UID: "ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:18.079995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.079971    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb" pod "ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" (UID: "ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7"). InnerVolumeSpecName "kube-api-access-z8trb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:18.084198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.084175    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:18.091401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.091378    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"]
Nov 01 09:00:18.178408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.178377    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv\" (UniqueName: \"kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv\") pod \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") "
Nov 01 09:00:18.178597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.178442    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca\") pod \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\" (UID: \"1f9841d8-5f7e-4743-b18d-71f37efd9ab0\") "
Nov 01 09:00:18.178669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.178603    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca\") pod \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") "
Nov 01 09:00:18.178773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.178759    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb\" (UniqueName: \"kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb\") pod \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\" (UID: \"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225\") "
Nov 01 09:00:18.178823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.178799    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca" pod "1f9841d8-5f7e-4743-b18d-71f37efd9ab0" (UID: "1f9841d8-5f7e-4743-b18d-71f37efd9ab0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:18.178964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.178945    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca" pod "bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" (UID: "bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:18.179301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.179277    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:18.179429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.179304    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:18.179429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.179320    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8trb\" (UniqueName: \"kubernetes.io/projected/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7-kube-api-access-z8trb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:18.179429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.179332    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:18.180480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.180459    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv" pod "1f9841d8-5f7e-4743-b18d-71f37efd9ab0" (UID: "1f9841d8-5f7e-4743-b18d-71f37efd9ab0"). InnerVolumeSpecName "kube-api-access-djthv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:18.180642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.180628    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb" pod "bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" (UID: "bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225"). InnerVolumeSpecName "kube-api-access-86vxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:18.280876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.280808    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vxb\" (UniqueName: \"kubernetes.io/projected/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225-kube-api-access-86vxb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:18.280876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.280836    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djthv\" (UniqueName: \"kubernetes.io/projected/1f9841d8-5f7e-4743-b18d-71f37efd9ab0-kube-api-access-djthv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:18.767388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.767351    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerStarted","Data":"f00c12cddfab94b43b4ac985dc9d7b305b3c18a3094a0455eb7e08c8f2c84795"}
Nov 01 09:00:18.768818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.768799    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"
Nov 01 09:00:18.768943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.768845    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x" event={"ID":"bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225","Type":"ContainerDied","Data":"8807a8fe8430d0991dd289575504f35efd2cd2d3085f1ce1de32439bd48a9a56"}
Nov 01 09:00:18.768943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.768899    4314 scope.go:122] "RemoveContainer" containerID="a9f8bc3cb749e96ec3705bd75fdb7ecaf58f78335ca3e038086dd9a0939666cd"
Nov 01 09:00:18.770414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.770393    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"
Nov 01 09:00:18.770414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.770404    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr" event={"ID":"ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7","Type":"ContainerDied","Data":"db318deebbc7f40054e94d1d47a53f59ae4b15758b35ba11ee4077563e510240"}
Nov 01 09:00:18.772655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.772633    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerStarted","Data":"c869f98d38da4389915ab95a102c242a7b4f2324eff12f40e7008bedbe1280d7"}
Nov 01 09:00:18.776117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.776094    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"
Nov 01 09:00:18.776217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.776099    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x" event={"ID":"1f9841d8-5f7e-4743-b18d-71f37efd9ab0","Type":"ContainerDied","Data":"17b14aa381e568cf8b9d1cef427293a45f2cfc711b1f656c6bc140aebde31dec"}
Nov 01 09:00:18.778753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.778108    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerStarted","Data":"b25cdcfd3344dc3579cc2e50d880902d6346ab1070b4fb8d776edccde81d2b5e"}
Nov 01 09:00:18.781684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.781667    4314 scope.go:122] "RemoveContainer" containerID="361219e2ca3344c04a8496118ffe54a77d6c31ff130a40daa03e15587753538c"
Nov 01 09:00:18.795714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.793938    4314 scope.go:122] "RemoveContainer" containerID="af3aa4e0a22e56e8d24ab2a1e904f2d96c0de1b5a18587c6fb169b1a8b634235"
Nov 01 09:00:18.925347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.925325    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:18.928925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:18.928893    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" path="/var/lib/kubelet/pods/9ce33414-ccd3-4630-acc2-81c87015071a/volumes"
Nov 01 09:00:19.091812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.091754    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652\" (UniqueName: \"kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652\") pod \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") "
Nov 01 09:00:19.091960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.091945    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca\") pod \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\" (UID: \"19eb33b6-4f4e-4823-b4d8-1506fea525c1\") "
Nov 01 09:00:19.092285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.092250    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca" pod "19eb33b6-4f4e-4823-b4d8-1506fea525c1" (UID: "19eb33b6-4f4e-4823-b4d8-1506fea525c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:19.093739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.093717    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652" pod "19eb33b6-4f4e-4823-b4d8-1506fea525c1" (UID: "19eb33b6-4f4e-4823-b4d8-1506fea525c1"). InnerVolumeSpecName "kube-api-access-5h652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:19.193316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.193281    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h652\" (UniqueName: \"kubernetes.io/projected/19eb33b6-4f4e-4823-b4d8-1506fea525c1-kube-api-access-5h652\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:19.193316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.193311    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19eb33b6-4f4e-4823-b4d8-1506fea525c1-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:19.202339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.202310    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"]
Nov 01 09:00:19.209877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.209854    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-03832c-29892060-ljfvr"]
Nov 01 09:00:19.420388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.420294    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"]
Nov 01 09:00:19.426099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.426072    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892060-6hc2x"]
Nov 01 09:00:19.527798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.527765    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"]
Nov 01 09:00:19.531938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.531913    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892060-hw84x"]
Nov 01 09:00:19.798286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.798245    4314 generic.go:356] "Generic (PLEG): container finished" podID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerID="c869f98d38da4389915ab95a102c242a7b4f2324eff12f40e7008bedbe1280d7" exitCode=60
Nov 01 09:00:19.798286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.798277    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerDied","Data":"c869f98d38da4389915ab95a102c242a7b4f2324eff12f40e7008bedbe1280d7"}
Nov 01 09:00:19.798840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.798395    4314 scope.go:122] "RemoveContainer" containerID="5b4781672f393de0fb8049621de74811c78e2fe23ab326d52a5b92ddb369c58b"
Nov 01 09:00:19.798958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.798939    4314 scope.go:122] "RemoveContainer" containerID="c869f98d38da4389915ab95a102c242a7b4f2324eff12f40e7008bedbe1280d7"
Nov 01 09:00:19.799343 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:19.799319    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn_odh-ai-gateway-infra(780e6ce7-e406-4e5d-af0b-875e24a32bc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4"
Nov 01 09:00:19.800156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.800139    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"
Nov 01 09:00:19.800156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.800149    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v" event={"ID":"19eb33b6-4f4e-4823-b4d8-1506fea525c1","Type":"ContainerDied","Data":"fb7184aa6d9f96c4007fc2fc03ae3feb4d6c9db655ac569c642db082abe21e71"}
Nov 01 09:00:19.803209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.803189    4314 generic.go:356] "Generic (PLEG): container finished" podID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerID="b25cdcfd3344dc3579cc2e50d880902d6346ab1070b4fb8d776edccde81d2b5e" exitCode=60
Nov 01 09:00:19.804828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.804803    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerDied","Data":"b25cdcfd3344dc3579cc2e50d880902d6346ab1070b4fb8d776edccde81d2b5e"}
Nov 01 09:00:19.805165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.805146    4314 scope.go:122] "RemoveContainer" containerID="b25cdcfd3344dc3579cc2e50d880902d6346ab1070b4fb8d776edccde81d2b5e"
Nov 01 09:00:19.805419 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:19.805396    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f_odh-ai-gateway-infra(e5e47389-7265-4de0-9cef-78d6bd2c1399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399"
Nov 01 09:00:19.805559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.805512    4314 generic.go:356] "Generic (PLEG): container finished" podID="c44466dc-a198-4995-86fb-2fc47266a514" containerID="f00c12cddfab94b43b4ac985dc9d7b305b3c18a3094a0455eb7e08c8f2c84795" exitCode=60
Nov 01 09:00:19.805616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.805576    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerDied","Data":"f00c12cddfab94b43b4ac985dc9d7b305b3c18a3094a0455eb7e08c8f2c84795"}
Nov 01 09:00:19.805830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.805817    4314 scope.go:122] "RemoveContainer" containerID="f00c12cddfab94b43b4ac985dc9d7b305b3c18a3094a0455eb7e08c8f2c84795"
Nov 01 09:00:19.806038 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:19.806022    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5_odh-ai-gateway-infra(c44466dc-a198-4995-86fb-2fc47266a514)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" podUID="c44466dc-a198-4995-86fb-2fc47266a514"
Nov 01 09:00:19.817598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.816719    4314 scope.go:122] "RemoveContainer" containerID="a553e16ce9b094c30a0eea22b4fa03ec455d89a5a8565894b36a79a611acde3c"
Nov 01 09:00:19.830317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.829602    4314 scope.go:122] "RemoveContainer" containerID="af2bebd3eca1dc6db5864b30423e7c69c062788a6a32178603987e42f119ed78"
Nov 01 09:00:19.845253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.845231    4314 scope.go:122] "RemoveContainer" containerID="318eb599d85f6fa29bc9b0826d78520cfbc42df0a68d235887e654b08b65ea3d"
Nov 01 09:00:19.890817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:19.890794    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"]
Nov 01 09:00:20.001671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.001636    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"]
Nov 01 09:00:20.231502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.231476    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"]
Nov 01 09:00:20.239876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.239852    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d0ba8f-29892060-xht7v"]
Nov 01 09:00:20.325568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.325522    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"]
Nov 01 09:00:20.940441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.940408    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" path="/var/lib/kubelet/pods/19eb33b6-4f4e-4823-b4d8-1506fea525c1/volumes"
Nov 01 09:00:20.941642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.941618    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" path="/var/lib/kubelet/pods/1f9841d8-5f7e-4743-b18d-71f37efd9ab0/volumes"
Nov 01 09:00:20.942349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.942330    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" path="/var/lib/kubelet/pods/ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7/volumes"
Nov 01 09:00:20.943515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.943496    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" path="/var/lib/kubelet/pods/bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225/volumes"
Nov 01 09:00:20.996283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:20.996252    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:21.077949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.077898    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:21.081382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.081361    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:21.118775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.118753    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca\") pod \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") "
Nov 01 09:00:21.118866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.118813    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v\" (UniqueName: \"kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v\") pod \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\" (UID: \"780e6ce7-e406-4e5d-af0b-875e24a32bc4\") "
Nov 01 09:00:21.119083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.119063    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca" pod "780e6ce7-e406-4e5d-af0b-875e24a32bc4" (UID: "780e6ce7-e406-4e5d-af0b-875e24a32bc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:21.119235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.119218    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e6ce7-e406-4e5d-af0b-875e24a32bc4-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:21.121113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.121089    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v" pod "780e6ce7-e406-4e5d-af0b-875e24a32bc4" (UID: "780e6ce7-e406-4e5d-af0b-875e24a32bc4"). InnerVolumeSpecName "kube-api-access-fmb2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:21.220441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.220417    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5\" (UniqueName: \"kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5\") pod \"c44466dc-a198-4995-86fb-2fc47266a514\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") "
Nov 01 09:00:21.220596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.220544    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca\") pod \"c44466dc-a198-4995-86fb-2fc47266a514\" (UID: \"c44466dc-a198-4995-86fb-2fc47266a514\") "
Nov 01 09:00:21.220596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.220569    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh\" (UniqueName: \"kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh\") pod \"e5e47389-7265-4de0-9cef-78d6bd2c1399\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") "
Nov 01 09:00:21.220711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.220638    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca\") pod \"e5e47389-7265-4de0-9cef-78d6bd2c1399\" (UID: \"e5e47389-7265-4de0-9cef-78d6bd2c1399\") "
Nov 01 09:00:21.221018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.220924    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca" pod "c44466dc-a198-4995-86fb-2fc47266a514" (UID: "c44466dc-a198-4995-86fb-2fc47266a514"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:21.221133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.221084    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca" pod "e5e47389-7265-4de0-9cef-78d6bd2c1399" (UID: "e5e47389-7265-4de0-9cef-78d6bd2c1399"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:21.221422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.221404    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44466dc-a198-4995-86fb-2fc47266a514-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:21.221513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.221426    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e47389-7265-4de0-9cef-78d6bd2c1399-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:21.221513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.221441    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmb2v\" (UniqueName: \"kubernetes.io/projected/780e6ce7-e406-4e5d-af0b-875e24a32bc4-kube-api-access-fmb2v\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:21.222889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.222868    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5" pod "c44466dc-a198-4995-86fb-2fc47266a514" (UID: "c44466dc-a198-4995-86fb-2fc47266a514"). InnerVolumeSpecName "kube-api-access-btzp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:21.222992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.222895    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh" pod "e5e47389-7265-4de0-9cef-78d6bd2c1399" (UID: "e5e47389-7265-4de0-9cef-78d6bd2c1399"). InnerVolumeSpecName "kube-api-access-xcxvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:21.322856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.322824    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btzp5\" (UniqueName: \"kubernetes.io/projected/c44466dc-a198-4995-86fb-2fc47266a514-kube-api-access-btzp5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:21.322856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.322853    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcxvh\" (UniqueName: \"kubernetes.io/projected/e5e47389-7265-4de0-9cef-78d6bd2c1399-kube-api-access-xcxvh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:21.835203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.835169    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"
Nov 01 09:00:21.835383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.835165    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn" event={"ID":"780e6ce7-e406-4e5d-af0b-875e24a32bc4","Type":"ContainerDied","Data":"8523eb75c1e057c96b0bec7ee4578e13f4111c6bb9d67159bc1bca5084f7e1eb"}
Nov 01 09:00:21.835383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.835369    4314 scope.go:122] "RemoveContainer" containerID="c869f98d38da4389915ab95a102c242a7b4f2324eff12f40e7008bedbe1280d7"
Nov 01 09:00:21.844189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.844156    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"
Nov 01 09:00:21.844321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.844209    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f" event={"ID":"e5e47389-7265-4de0-9cef-78d6bd2c1399","Type":"ContainerDied","Data":"8bf4723d59076aadbf6de6667a543f39e606c40c1cf1147c1910577c0301465b"}
Nov 01 09:00:21.847469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.847435    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5" event={"ID":"c44466dc-a198-4995-86fb-2fc47266a514","Type":"ContainerDied","Data":"9edc6f644dd1072383b2a658feb2fd98c0751bb5a4d26f9beb2d5aab6acf93b3"}
Nov 01 09:00:21.847562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.847513    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"
Nov 01 09:00:21.855940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.855925    4314 scope.go:122] "RemoveContainer" containerID="b25cdcfd3344dc3579cc2e50d880902d6346ab1070b4fb8d776edccde81d2b5e"
Nov 01 09:00:21.865142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.865126    4314 scope.go:122] "RemoveContainer" containerID="f00c12cddfab94b43b4ac985dc9d7b305b3c18a3094a0455eb7e08c8f2c84795"
Nov 01 09:00:21.992522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:21.992492    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"]
Nov 01 09:00:22.037606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.037578    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892060-9vz8f"]
Nov 01 09:00:22.101914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.101851    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"]
Nov 01 09:00:22.106005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.105980    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3901b8-29892060-pmfl5"]
Nov 01 09:00:22.215432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.215401    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"]
Nov 01 09:00:22.221998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.221972    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892060-qq8wn"]
Nov 01 09:00:22.378527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.378442    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:22.646938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.646835    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:22.814312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.814282    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:22.929104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.929026    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" path="/var/lib/kubelet/pods/780e6ce7-e406-4e5d-af0b-875e24a32bc4/volumes"
Nov 01 09:00:22.929522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.929501    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44466dc-a198-4995-86fb-2fc47266a514" path="/var/lib/kubelet/pods/c44466dc-a198-4995-86fb-2fc47266a514/volumes"
Nov 01 09:00:22.929971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:22.929952    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" path="/var/lib/kubelet/pods/e5e47389-7265-4de0-9cef-78d6bd2c1399/volumes"
Nov 01 09:00:23.123254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:23.123219    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:28.308209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:28.308177    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:28.678372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:28.678283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:29.521668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:29.521636    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"]
Nov 01 09:00:29.522087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:29.521835    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy" containerID="cri-o://beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64" gracePeriod=30
Nov 01 09:00:29.568115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:29.568081    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"]
Nov 01 09:00:29.568328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:29.568305    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy" containerID="cri-o://0fcd56093e21a675a1c16afefb97c14f084db6044d1127a0cfd254d0d93169a8" gracePeriod=30
Nov 01 09:00:33.328880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:33.328838    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:33.490668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:33.490635    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:33.731307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:33.731268    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:34.297739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.297704    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:34.917850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.917817    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:34.935227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.935200    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 09:00:34.959288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.959238    4314 generic.go:356] "Generic (PLEG): container finished" podID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerID="0fcd56093e21a675a1c16afefb97c14f084db6044d1127a0cfd254d0d93169a8" exitCode=0
Nov 01 09:00:34.959813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.959773    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" event={"ID":"012254b6-ea0f-4d47-a50e-80da612a43f1","Type":"ContainerDied","Data":"0fcd56093e21a675a1c16afefb97c14f084db6044d1127a0cfd254d0d93169a8"}
Nov 01 09:00:34.962226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.962166    4314 generic.go:356] "Generic (PLEG): container finished" podID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerID="beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64" exitCode=0
Nov 01 09:00:34.962542 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.962494    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" event={"ID":"cbf2af26-0a56-4830-8a91-cfe35c4c1023","Type":"ContainerDied","Data":"beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64"}
Nov 01 09:00:34.963084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.962709    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7" event={"ID":"cbf2af26-0a56-4830-8a91-cfe35c4c1023","Type":"ContainerDied","Data":"2c194e5bec1fb20b01521213ddeb16692898b66338d2cec9bd16f30944371ec8"}
Nov 01 09:00:34.963084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.962770    4314 scope.go:122] "RemoveContainer" containerID="beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64"
Nov 01 09:00:34.963084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:34.962924    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020062    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020204    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020249    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020297    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020338    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020403    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020434    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020495    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020549    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020662    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert\") pod \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\" (UID: \"cbf2af26-0a56-4830-8a91-cfe35c4c1023\") "
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.020747    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.021262    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.021283    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.028893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.023769    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.069956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.054685    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.104563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.096672    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:35.104563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.097092    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.104563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.097899    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "kube-api-access-l24fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:35.104563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.099764    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:35.126233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.126194    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert" pod "cbf2af26-0a56-4830-8a91-cfe35c4c1023" (UID: "cbf2af26-0a56-4830-8a91-cfe35c4c1023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:35.136660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.134335    4314 scope.go:122] "RemoveContainer" containerID="beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64"
Nov 01 09:00:35.137009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.136491    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24fv\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-kube-api-access-l24fv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.137009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.136969    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.137331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.136988    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.137331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.137276    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.137331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.137294    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.137331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.137308    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.138089 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:35.138042    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64\": container with ID starting with beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64 not found: ID does not exist" containerID="beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64"
Nov 01 09:00:35.138201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.138098    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64"} err="failed to get container status \"beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64\": rpc error: code = NotFound desc = could not find container \"beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64\": container with ID starting with beaabaecbfb7230b59e87bd58890d62d53d91d390c733bfa71d230bd0a913c64 not found: ID does not exist"
Nov 01 09:00:35.161968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.161854    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 09:00:35.238881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.238854    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.238912    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.238957    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239019    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239058    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239087    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239197    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.239295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239236    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239309    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239370    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket\") pod \"012254b6-ea0f-4d47-a50e-80da612a43f1\" (UID: \"012254b6-ea0f-4d47-a50e-80da612a43f1\") "
Nov 01 09:00:35.239515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239480    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.239702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239645    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.239830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239747    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.239830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.239769    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:35.240147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.240125    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.240229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.240154    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.240229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.240170    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.240229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.240186    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012254b6-ea0f-4d47-a50e-80da612a43f1-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.240229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.240202    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf2af26-0a56-4830-8a91-cfe35c4c1023-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.240229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.240216    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.241479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.241443    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:35.241479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.241459    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.241641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.241512    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:35.241641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.241522    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6" pod "012254b6-ea0f-4d47-a50e-80da612a43f1" (UID: "012254b6-ea0f-4d47-a50e-80da612a43f1"). InnerVolumeSpecName "kube-api-access-vrzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:35.281773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.281724    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:35.348806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.348709    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.348806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.348755    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.348806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.348774    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrzt6\" (UniqueName: \"kubernetes.io/projected/012254b6-ea0f-4d47-a50e-80da612a43f1-kube-api-access-vrzt6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.348806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.348791    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012254b6-ea0f-4d47-a50e-80da612a43f1-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.403798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.403761    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"]
Nov 01 09:00:35.407278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.407255    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-ptzc7"]
Nov 01 09:00:35.971546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.971510    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"
Nov 01 09:00:35.972027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.971512    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q" event={"ID":"012254b6-ea0f-4d47-a50e-80da612a43f1","Type":"ContainerDied","Data":"59d553452f301a9642e2e0fd99607249ffecd877c291fa2ce5f7dd2e1b3c5183"}
Nov 01 09:00:35.972027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:35.971683    4314 scope.go:122] "RemoveContainer" containerID="0fcd56093e21a675a1c16afefb97c14f084db6044d1127a0cfd254d0d93169a8"
Nov 01 09:00:36.127550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:36.127503    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"]
Nov 01 09:00:36.133497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:36.133463    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-48k6q"]
Nov 01 09:00:36.930977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:36.930940    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" path="/var/lib/kubelet/pods/012254b6-ea0f-4d47-a50e-80da612a43f1/volumes"
Nov 01 09:00:36.931479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:36.931462    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" path="/var/lib/kubelet/pods/cbf2af26-0a56-4830-8a91-cfe35c4c1023/volumes"
Nov 01 09:00:38.546377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:38.546347    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:39.991848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:39.991814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:40.260148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:40.260021    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:40.424130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:40.424097    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:41.322317 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:00:41.322287    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:00:42.409368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409335    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"]
Nov 01 09:00:42.409930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409914    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409931    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409944    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409949    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409957    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409964    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409972    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy"
Nov 01 09:00:42.409978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409976    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409983    4314 cpu_manager.go:414] "containerMap: removing container" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409988    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.409995    4314 cpu_manager.go:414] "containerMap: removing container" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410001    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410008    4314 cpu_manager.go:414] "containerMap: removing container" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410012    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410021    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410025    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410032    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410037    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410043    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410047    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410060    4314 cpu_manager.go:414] "containerMap: removing container" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410065    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410074    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410078    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410085    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410089    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410100    4314 cpu_manager.go:414] "containerMap: removing container" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410104    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410110    4314 cpu_manager.go:414] "containerMap: removing container" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410114    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410121    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410127    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410135    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410139    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410147    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410151    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410159    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410164    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410169    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410173    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410183    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410187    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410197    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.410191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410201    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410277    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410283    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410291    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410298    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6432463-4ec0-4323-98e8-4ea25e21c415" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410304    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410310    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410315    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410319    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410324    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410331    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410336    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf2af26-0a56-4830-8a91-cfe35c4c1023" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410341    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410345    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410350    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="35061eb7-9af6-46fd-a0f1-f3358f762de6" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410355    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d694432c-21e5-4c92-be3f-d4674d18b15b" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410361    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410366    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410374    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44ff955-8bb9-481d-94d1-377fdb20961d" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410380    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410385    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="012254b6-ea0f-4d47-a50e-80da612a43f1" containerName="istio-proxy"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410390    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410396    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:00:42.411237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.410400    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:00:42.415329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.415313    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.449801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.449767    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.449937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.449830    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.449990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.449956    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9gf\" (UniqueName: \"kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.468084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.468056    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"]
Nov 01 09:00:42.509082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.509055    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:00:42.509082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.509056    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:00:42.509082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.509079    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:00:42.510325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.510311    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:00:42.551573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.551515    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.551709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.551656    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9gf\" (UniqueName: \"kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.551810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.551792    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.552144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.552126    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.554092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.554069    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.608702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.608672    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9gf\" (UniqueName: \"kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf\") pod \"maas-api-revoke-sub-287ebe29-l8dpk\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:42.729213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:42.729128    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:43.105304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:43.105270    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"]
Nov 01 09:00:43.106349 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:00:43.106325    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bf487d_ec2a_4038_9337_51331994ef7c.slice/crio-0a10c27bde1e86c8d2d3c1849260f951dd715c5a4261d22d3445a48d383c5b86 WatchSource:0}: Error finding container 0a10c27bde1e86c8d2d3c1849260f951dd715c5a4261d22d3445a48d383c5b86: Status 404 returned error can't find the container with id 0a10c27bde1e86c8d2d3c1849260f951dd715c5a4261d22d3445a48d383c5b86
Nov 01 09:00:43.200228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:43.200197    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:43.374216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:43.371400    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:44.035590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:44.035524    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:44.036511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:44.036481    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk" event={"ID":"97bf487d-ec2a-4038-9337-51331994ef7c","Type":"ContainerStarted","Data":"4532d55302bdd459ffe26f83b17a3dbb958361cc9e0b0d5d9de793be6a9baf61"}
Nov 01 09:00:44.036710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:44.036554    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk" event={"ID":"97bf487d-ec2a-4038-9337-51331994ef7c","Type":"ContainerStarted","Data":"0a10c27bde1e86c8d2d3c1849260f951dd715c5a4261d22d3445a48d383c5b86"}
Nov 01 09:00:45.046571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.046520    4314 generic.go:356] "Generic (PLEG): container finished" podID="97bf487d-ec2a-4038-9337-51331994ef7c" containerID="4532d55302bdd459ffe26f83b17a3dbb958361cc9e0b0d5d9de793be6a9baf61" exitCode=0
Nov 01 09:00:45.046571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.046564    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk" event={"ID":"97bf487d-ec2a-4038-9337-51331994ef7c","Type":"ContainerDied","Data":"4532d55302bdd459ffe26f83b17a3dbb958361cc9e0b0d5d9de793be6a9baf61"}
Nov 01 09:00:45.051380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.051344    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"]
Nov 01 09:00:45.125732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.125680    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db8a280-hhmls"]
Nov 01 09:00:45.368606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.368365    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:45.433956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.433925    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca\") pod \"97bf487d-ec2a-4038-9337-51331994ef7c\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") "
Nov 01 09:00:45.434158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.433998    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca\") pod \"97bf487d-ec2a-4038-9337-51331994ef7c\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") "
Nov 01 09:00:45.434158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.434154    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf\" (UniqueName: \"kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf\") pod \"97bf487d-ec2a-4038-9337-51331994ef7c\" (UID: \"97bf487d-ec2a-4038-9337-51331994ef7c\") "
Nov 01 09:00:45.434368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.434346    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca" pod "97bf487d-ec2a-4038-9337-51331994ef7c" (UID: "97bf487d-ec2a-4038-9337-51331994ef7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:45.434713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.434685    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:45.436799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.436773    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf" pod "97bf487d-ec2a-4038-9337-51331994ef7c" (UID: "97bf487d-ec2a-4038-9337-51331994ef7c"). InnerVolumeSpecName "kube-api-access-dr9gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:45.436891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.436833    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca" pod "97bf487d-ec2a-4038-9337-51331994ef7c" (UID: "97bf487d-ec2a-4038-9337-51331994ef7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:00:45.535463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.535425    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9gf\" (UniqueName: \"kubernetes.io/projected/97bf487d-ec2a-4038-9337-51331994ef7c-kube-api-access-dr9gf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:45.535463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.535462    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97bf487d-ec2a-4038-9337-51331994ef7c-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:45.698937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.698851    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:45.848850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:45.848815    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:46.055817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:46.055787    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"
Nov 01 09:00:46.056225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:46.055822    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk" event={"ID":"97bf487d-ec2a-4038-9337-51331994ef7c","Type":"ContainerDied","Data":"0a10c27bde1e86c8d2d3c1849260f951dd715c5a4261d22d3445a48d383c5b86"}
Nov 01 09:00:46.056225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:46.055877    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a10c27bde1e86c8d2d3c1849260f951dd715c5a4261d22d3445a48d383c5b86"
Nov 01 09:00:46.929056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:46.929023    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9efc96-3391-462c-8dae-96263874ae42" path="/var/lib/kubelet/pods/eb9efc96-3391-462c-8dae-96263874ae42/volumes"
Nov 01 09:00:49.300594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:49.300560    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:54.199412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:54.199377    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:58.933560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:58.933487    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:59.096154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:59.096122    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:00:59.274559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:00:59.274492    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:03.055387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:03.055350    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:03.687645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:03.687601    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:04.010697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:04.010664    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:04.166548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:04.166505    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:07.756741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.756709    4314 scope.go:122] "RemoveContainer" containerID="0b007855f5b6e12ce11bff60feb62ef2410a8e22a444bf6757db3cc50543b66f"
Nov 01 09:01:07.771026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.771000    4314 scope.go:122] "RemoveContainer" containerID="8dcc95616f011c0aa82195352ce888e2b821017c231088ec0d54a5b2b8654a41"
Nov 01 09:01:07.780979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.780956    4314 scope.go:122] "RemoveContainer" containerID="b159f6a4b65c988ab4242d24e54ce7c3010ba35944b18ad777748b6d7ad47a18"
Nov 01 09:01:07.791212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.791186    4314 scope.go:122] "RemoveContainer" containerID="277d51acc216418e37ec5daa7994d8a83f4cba406c79ee55f499ea19615c3e12"
Nov 01 09:01:07.800363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.800338    4314 scope.go:122] "RemoveContainer" containerID="b629e620b7f86626acf1b13566b29ab8b773c4d3dbcb0f3dd3d0544c9a99be03"
Nov 01 09:01:07.809316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.809297    4314 scope.go:122] "RemoveContainer" containerID="c5d92945c877b183a10b10223813d6f884ec4df12ec81e204d27f196ee2d8552"
Nov 01 09:01:07.817609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.817587    4314 scope.go:122] "RemoveContainer" containerID="5255bf2e4900f3fce4021e82c1c16c330f8ebceb98e176c16f2e23527cd59904"
Nov 01 09:01:07.826209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:07.826191    4314 scope.go:122] "RemoveContainer" containerID="300e3f582a794132685b22ae0088bd4b08e77ef04d10341bbaf2e31a87929e64"
Nov 01 09:01:08.022209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:08.022129    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:08.207753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:08.207718    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:08.356822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:08.356731    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:09.084396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:09.084360    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:09.769172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:09.769135    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:11.166368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:11.166334    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:11.331129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:11.331093    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:11.371124 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:01:11.371073    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:01:13.559869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:13.559837    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:15.384325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:15.384289    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:18.046835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:18.046802    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"]
Nov 01 09:01:18.092916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:18.092884    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53b0f97-7d6nc"]
Nov 01 09:01:18.929609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:18.929516    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdc4e30-398f-4f24-b754-7c97c0284877" path="/var/lib/kubelet/pods/1fdc4e30-398f-4f24-b754-7c97c0284877/volumes"
Nov 01 09:01:20.178981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:20.178945    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:21.462282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:21.462246    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:21.605547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:21.605511    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:23.028412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:23.028378    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:23.501635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:23.501527    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"]
Nov 01 09:01:23.501865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:23.501842    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy" containerID="cri-o://2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0" gracePeriod=30
Nov 01 09:01:23.506716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:23.506694    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:24.449810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:24.449768    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:25.100699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:25.100650    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:25.335075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:25.335038    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:25.616466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:25.616432    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:28.785767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.785740    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 09:01:28.888867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.888825    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.888883    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.888921    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.888956    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889018    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889119    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889161    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:01:28.889288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889226    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889287    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889445    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket\") pod \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\" (UID: \"05119dac-77cf-47aa-ace8-b97cc03dfb1e\") "
Nov 01 09:01:28.889491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889456    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:01:28.889801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889778    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:01:28.889888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.889805    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:01:28.890072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.890050    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:01:28.890206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.890187    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.890250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.890214    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.890250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.890230    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.890250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.890244    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.891423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.891394    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:01:28.891574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.891418    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:01:28.891690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.891668    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "kube-api-access-9kv5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:01:28.891799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.891700    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy" pod "05119dac-77cf-47aa-ace8-b97cc03dfb1e" (UID: "05119dac-77cf-47aa-ace8-b97cc03dfb1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:01:28.992867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.992824    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kv5r\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-kube-api-access-9kv5r\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.992867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.992870    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.993108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.992888    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.993108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.992910    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05119dac-77cf-47aa-ace8-b97cc03dfb1e-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:28.993108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:28.992925    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05119dac-77cf-47aa-ace8-b97cc03dfb1e-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:01:29.280526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.280485    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:29.355932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.355900    4314 generic.go:356] "Generic (PLEG): container finished" podID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerID="2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0" exitCode=0
Nov 01 09:01:29.356104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.355964    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"
Nov 01 09:01:29.356104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.355990    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" event={"ID":"05119dac-77cf-47aa-ace8-b97cc03dfb1e","Type":"ContainerDied","Data":"2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0"}
Nov 01 09:01:29.356104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.356079    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf" event={"ID":"05119dac-77cf-47aa-ace8-b97cc03dfb1e","Type":"ContainerDied","Data":"17389c3a8b1cb8ea83f8cc5a3bfc510167a153c38703b9e432497ae3d9eb31b9"}
Nov 01 09:01:29.356104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.356103    4314 scope.go:122] "RemoveContainer" containerID="2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0"
Nov 01 09:01:29.366146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.366127    4314 scope.go:122] "RemoveContainer" containerID="e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"
Nov 01 09:01:29.375221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.375193    4314 scope.go:122] "RemoveContainer" containerID="2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0"
Nov 01 09:01:29.375513 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:01:29.375493    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0\": container with ID starting with 2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0 not found: ID does not exist" containerID="2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0"
Nov 01 09:01:29.375600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.375524    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0"} err="failed to get container status \"2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0\": rpc error: code = NotFound desc = could not find container \"2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0\": container with ID starting with 2161a932b79e6455b480996b5a55ea3d861dd6155d2e6a05ebf9e452b012fef0 not found: ID does not exist"
Nov 01 09:01:29.375600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.375572    4314 scope.go:122] "RemoveContainer" containerID="e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"
Nov 01 09:01:29.375841 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:01:29.375823    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6\": container with ID starting with e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6 not found: ID does not exist" containerID="e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"
Nov 01 09:01:29.375883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.375848    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6"} err="failed to get container status \"e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6\": rpc error: code = NotFound desc = could not find container \"e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6\": container with ID starting with e5f3e0568f722787767caf8aac726353b355b6224fd33477b4a96ac1990b03b6 not found: ID does not exist"
Nov 01 09:01:29.450651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.450617    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:29.515209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.515176    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"]
Nov 01 09:01:29.560339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.560250    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d0ba8f-openshift-default-5dd99b8885-ftqsf"]
Nov 01 09:01:29.607058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:29.607027    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:30.443393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:30.443352    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:30.594591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:30.594527    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:30.750885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:30.750803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:30.929100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:30.929062    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" path="/var/lib/kubelet/pods/05119dac-77cf-47aa-ace8-b97cc03dfb1e/volumes"
Nov 01 09:01:35.961028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:35.960942    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:37.904799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:37.904763    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:38.076836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.076797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:38.311361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311319    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"]
Nov 01 09:01:38.311937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311920    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311939    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311955    4314 cpu_manager.go:414] "containerMap: removing container" podUID="97bf487d-ec2a-4038-9337-51331994ef7c" containerName="revoke-keys"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311960    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97bf487d-ec2a-4038-9337-51331994ef7c" containerName="revoke-keys"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311970    4314 cpu_manager.go:414] "containerMap: removing container" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311975    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311982    4314 cpu_manager.go:414] "containerMap: removing container" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.311986    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:01:38.312000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312002    4314 cpu_manager.go:414] "containerMap: removing container" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312007    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19eb33b6-4f4e-4823-b4d8-1506fea525c1" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312023    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312030    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312049    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312057    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e47389-7265-4de0-9cef-78d6bd2c1399" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312070    4314 cpu_manager.go:414] "containerMap: removing container" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312074    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312083    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312087    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312096    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312101    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312189    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9841d8-5f7e-4743-b18d-71f37efd9ab0" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312199    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="97bf487d-ec2a-4038-9337-51331994ef7c" containerName="revoke-keys"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312204    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312213    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="780e6ce7-e406-4e5d-af0b-875e24a32bc4" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312221    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaf3c37-a1a0-4b86-9e45-b4b7f2a78225" containerName="cleanup"
Nov 01 09:01:38.312227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312226    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44466dc-a198-4995-86fb-2fc47266a514" containerName="cleanup"
Nov 01 09:01:38.312782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312235    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce33414-ccd3-4630-acc2-81c87015071a" containerName="istio-proxy"
Nov 01 09:01:38.312782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312242    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6f5f91-c9d1-404b-bc75-6dbd7d03e7f7" containerName="cleanup"
Nov 01 09:01:38.312782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.312247    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="05119dac-77cf-47aa-ace8-b97cc03dfb1e" containerName="istio-proxy"
Nov 01 09:01:38.317311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.317294    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.318607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.318583    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.318731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.318621    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd68f\" (UniqueName: \"kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f\") pod \"payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.368922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.368892    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-991c28-dockercfg-66whl\""
Nov 01 09:01:38.369237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.369217    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-991c28\""
Nov 01 09:01:38.430588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.411421    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"]
Nov 01 09:01:38.458552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.449783    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.458552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.449846    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd68f\" (UniqueName: \"kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f\") pod \"payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.458552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.451714    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.496153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.492836    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"]
Nov 01 09:01:38.519609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.501816    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.540992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.540942    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd68f\" (UniqueName: \"kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f\") pod \"payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.549410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.548956    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"]
Nov 01 09:01:38.633310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.633225    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:38.653269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.653218    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.653448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.653366    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wg7\" (UniqueName: \"kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7\") pod \"payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.754603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.754567    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wg7\" (UniqueName: \"kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7\") pod \"payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.754868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.754849    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.755828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.755803    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.811746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.811713    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wg7\" (UniqueName: \"kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7\") pod \"payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:38.866011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:38.865979    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:39.012420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.012392    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"]
Nov 01 09:01:39.013656 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:01:39.013632    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28402fdc_4c72_4a12_ad1a_04c567a29a0e.slice/crio-608aecfe900642050a4681a74bd63df09a7663ec2cd4ec37907dd8d8fceca5a1 WatchSource:0}: Error finding container 608aecfe900642050a4681a74bd63df09a7663ec2cd4ec37907dd8d8fceca5a1: Status 404 returned error can't find the container with id 608aecfe900642050a4681a74bd63df09a7663ec2cd4ec37907dd8d8fceca5a1
Nov 01 09:01:39.234441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.234414    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"]
Nov 01 09:01:39.235160 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:01:39.235134    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb9c8e5_9634_4020_8e3f_febedab10fa1.slice/crio-bb44a2607f97b0193969b1d83763536887b3af76a9264af278ad9d50a3262bfc WatchSource:0}: Error finding container bb44a2607f97b0193969b1d83763536887b3af76a9264af278ad9d50a3262bfc: Status 404 returned error can't find the container with id bb44a2607f97b0193969b1d83763536887b3af76a9264af278ad9d50a3262bfc
Nov 01 09:01:39.547128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.547037    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd" event={"ID":"28402fdc-4c72-4a12-ad1a-04c567a29a0e","Type":"ContainerStarted","Data":"318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414"}
Nov 01 09:01:39.547128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.547099    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd" event={"ID":"28402fdc-4c72-4a12-ad1a-04c567a29a0e","Type":"ContainerStarted","Data":"608aecfe900642050a4681a74bd63df09a7663ec2cd4ec37907dd8d8fceca5a1"}
Nov 01 09:01:39.547340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.547132    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:39.548611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.548582    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h" event={"ID":"1eb9c8e5-9634-4020-8e3f-febedab10fa1","Type":"ContainerStarted","Data":"61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908"}
Nov 01 09:01:39.548739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.548625    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h" event={"ID":"1eb9c8e5-9634-4020-8e3f-febedab10fa1","Type":"ContainerStarted","Data":"bb44a2607f97b0193969b1d83763536887b3af76a9264af278ad9d50a3262bfc"}
Nov 01 09:01:39.548739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.548665    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:39.655075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.655010    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd" podStartSLOduration=1.654994178 podStartE2EDuration="1.654994178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:39.65479967 +0000 UTC m=+1772.849194219" watchObservedRunningTime="2026-11-01 09:01:39.654994178 +0000 UTC m=+1772.849388734"
Nov 01 09:01:39.778497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:39.775934    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h" podStartSLOduration=1.7758843450000001 podStartE2EDuration="1.775884345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:39.762861663 +0000 UTC m=+1772.957256241" watchObservedRunningTime="2026-11-01 09:01:39.775884345 +0000 UTC m=+1772.970278900"
Nov 01 09:01:40.776647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:40.776608    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:41.322236 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:01:41.322195    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:01:41.407864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:41.407828    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:45.827111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:45.827069    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:46.724451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:46.724416    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:50.558139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:50.558105    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:01:50.558646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:50.558222    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:01:51.575991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:51.575941    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:01:56.345438 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:01:56.345398    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:01.186598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:01.186560    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:02.831348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:02.831310    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:05.172057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:05.172018    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:06.219145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:06.219100    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:06.376483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:06.376444    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:06.523257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:06.523224    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:07.946866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:07.946833    4314 scope.go:122] "RemoveContainer" containerID="0b3713a554439603cd745b9dcd440cbc9722f389b0bb78bd8025351bff42a8e8"
Nov 01 09:02:10.226057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:10.226019    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:11.340790 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:11.340749    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:02:11.823765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:11.823734    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:12.870781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:12.870746    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"]
Nov 01 09:02:13.217140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:13.217051    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:13.373658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:13.373612    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:14.257559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.256906    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"]
Nov 01 09:02:14.257559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.257158    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" containerID="cri-o://9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0" gracePeriod=30
Nov 01 09:02:14.306348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.306312    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"]
Nov 01 09:02:14.306980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.306938    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy" containerID="cri-o://04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a" gracePeriod=30
Nov 01 09:02:14.318387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.318331    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"]
Nov 01 09:02:14.318682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.318648    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" containerID="cri-o://33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263" gracePeriod=30
Nov 01 09:02:14.348028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.347996    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"]
Nov 01 09:02:14.348289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:14.348236    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" containerID="cri-o://342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61" gracePeriod=30
Nov 01 09:02:15.886992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:15.886953    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-lnjmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:02:15.887475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:15.887016    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:02:16.378437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:16.378399    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:17.496028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:17.495994    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:17.871703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:17.871595    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx" podUID="b1bbc18c-3927-4686-9faf-5147d4ccb398" containerName="payload-processing" containerID="cri-o://cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54" gracePeriod=25
Nov 01 09:02:18.146885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.146857    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 09:02:18.226106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.226064    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume\") pod \"b1bbc18c-3927-4686-9faf-5147d4ccb398\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") "
Nov 01 09:02:18.226295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.226161    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf\" (UniqueName: \"kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf\") pod \"b1bbc18c-3927-4686-9faf-5147d4ccb398\" (UID: \"b1bbc18c-3927-4686-9faf-5147d4ccb398\") "
Nov 01 09:02:18.226516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.226489    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume" pod "b1bbc18c-3927-4686-9faf-5147d4ccb398" (UID: "b1bbc18c-3927-4686-9faf-5147d4ccb398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:18.226821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.226806    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bbc18c-3927-4686-9faf-5147d4ccb398-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:18.228299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.228279    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf" pod "b1bbc18c-3927-4686-9faf-5147d4ccb398" (UID: "b1bbc18c-3927-4686-9faf-5147d4ccb398"). InnerVolumeSpecName "kube-api-access-nt5mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:18.328234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.328198    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5mf\" (UniqueName: \"kubernetes.io/projected/b1bbc18c-3927-4686-9faf-5147d4ccb398-kube-api-access-nt5mf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:18.821619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.821586    4314 generic.go:356] "Generic (PLEG): container finished" podID="b1bbc18c-3927-4686-9faf-5147d4ccb398" containerID="cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54" exitCode=0
Nov 01 09:02:18.822085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.821633    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"
Nov 01 09:02:18.822085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.821676    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx" event={"ID":"b1bbc18c-3927-4686-9faf-5147d4ccb398","Type":"ContainerDied","Data":"cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54"}
Nov 01 09:02:18.822085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.821732    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx" event={"ID":"b1bbc18c-3927-4686-9faf-5147d4ccb398","Type":"ContainerDied","Data":"156b857ca0edcef008aa84b2f7cbe3c521bbc3fdaa8554c8656dcd4ccab36c1f"}
Nov 01 09:02:18.822085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.821755    4314 scope.go:122] "RemoveContainer" containerID="cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54"
Nov 01 09:02:18.837158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.837126    4314 scope.go:122] "RemoveContainer" containerID="cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54"
Nov 01 09:02:18.837767 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:18.837734    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54\": container with ID starting with cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54 not found: ID does not exist" containerID="cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54"
Nov 01 09:02:18.837911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.837774    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54"} err="failed to get container status \"cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54\": rpc error: code = NotFound desc = could not find container \"cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54\": container with ID starting with cf8aaff75d2014af323cd52268d0d41b15d8d1d47320a86e6b35046de5cf9b54 not found: ID does not exist"
Nov 01 09:02:18.975952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:18.975909    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"]
Nov 01 09:02:19.022403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.022366    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03832c-5c6489c5df-r47gx"]
Nov 01 09:02:19.122787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.122705    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-sdwlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:02:19.122787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.122762    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:02:19.208071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.208031    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tfphq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:02:19.208242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.208104    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:02:19.660610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.650997    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 09:02:19.681488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.681454    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 09:02:19.699320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.699290    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 09:02:19.718249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.718221    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 09:02:19.750085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750047    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.750254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750137    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.750254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750186    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.750254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750220    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.750451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750254    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.750451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750284    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.750451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750320    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.750451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750328    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.750451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750422    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750453    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750488    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750527    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750586    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750596    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750633    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750668    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.750768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750719    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750816    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750827    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750888    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750929    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750956    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750956    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.750993    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.751179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.751046    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.751636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.751209    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753123    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753134    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753176    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753397    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753455    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753459    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753492    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.753596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753495    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "kube-api-access-57z6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753599    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "kube-api-access-m5rw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753602    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753656    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753704    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753721    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753761    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753807    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753848    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753874    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753901    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753928    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.753950    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754011    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754047    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket\") pod \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\" (UID: \"119bbcf8-3604-44d5-918c-7b5c5f10aabd\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754089    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket\") pod \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\" (UID: \"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754120    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754108    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754145    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754149    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754199    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754306    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d\") pod \"bf477d9c-9a2f-4618-998a-150e6baa9147\" (UID: \"bf477d9c-9a2f-4618-998a-150e6baa9147\") "
Nov 01 09:02:19.754328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754340    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.754343    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert\") pod \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\" (UID: \"4636a672-4b3c-4f31-b8b6-3eef2dde47db\") "
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755085    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755110    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755126    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755142    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755158    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755173    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755306    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755322    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755337    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755353    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755365    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755380    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755395    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf477d9c-9a2f-4618-998a-150e6baa9147-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755409    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755422    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57z6t\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-kube-api-access-57z6t\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755435    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5rw5\" (UniqueName: \"kubernetes.io/projected/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-kube-api-access-m5rw5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755448    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755460    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755469    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755477    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf477d9c-9a2f-4618-998a-150e6baa9147-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.755526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755485    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.756864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755646    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.756864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.755849    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.756864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.756718    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.757019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.756976    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.757019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757000    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "kube-api-access-nd487". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.757239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757191    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.757351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757239    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.757417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757361    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy" pod "119bbcf8-3604-44d5-918c-7b5c5f10aabd" (UID: "119bbcf8-3604-44d5-918c-7b5c5f10aabd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.757417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757368    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.757578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757456    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.757633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757604    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:19.757834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757816    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo" pod "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" (UID: "2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:02:19.757942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.757925    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy" pod "4636a672-4b3c-4f31-b8b6-3eef2dde47db" (UID: "4636a672-4b3c-4f31-b8b6-3eef2dde47db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:19.758122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.758101    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:02:19.759096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.759075    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d" pod "bf477d9c-9a2f-4618-998a-150e6baa9147" (UID: "bf477d9c-9a2f-4618-998a-150e6baa9147"). InnerVolumeSpecName "kube-api-access-pqc9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:19.829777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.829739    4314 generic.go:356] "Generic (PLEG): container finished" podID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerID="9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0" exitCode=0
Nov 01 09:02:19.830283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.829796    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"
Nov 01 09:02:19.830283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.829821    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" event={"ID":"bf477d9c-9a2f-4618-998a-150e6baa9147","Type":"ContainerDied","Data":"9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0"}
Nov 01 09:02:19.830283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.829871    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj" event={"ID":"bf477d9c-9a2f-4618-998a-150e6baa9147","Type":"ContainerDied","Data":"3fe9b84c2c494c46a0d9f69fa8d431365d83287e0bef4f201c5f3e72d1b37ce6"}
Nov 01 09:02:19.830283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.829899    4314 scope.go:122] "RemoveContainer" containerID="9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0"
Nov 01 09:02:19.831480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.831457    4314 generic.go:356] "Generic (PLEG): container finished" podID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerID="33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263" exitCode=0
Nov 01 09:02:19.831615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.831594    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" event={"ID":"119bbcf8-3604-44d5-918c-7b5c5f10aabd","Type":"ContainerDied","Data":"33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263"}
Nov 01 09:02:19.831615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.831608    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"
Nov 01 09:02:19.831745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.831640    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm" event={"ID":"119bbcf8-3604-44d5-918c-7b5c5f10aabd","Type":"ContainerDied","Data":"315695bcd02ce537549b926b3dcf765d6aa1a2220fff9863567eb2019111501a"}
Nov 01 09:02:19.837570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.833171    4314 generic.go:356] "Generic (PLEG): container finished" podID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerID="04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a" exitCode=0
Nov 01 09:02:19.837570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.833235    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"
Nov 01 09:02:19.837570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.833254    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" event={"ID":"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b","Type":"ContainerDied","Data":"04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a"}
Nov 01 09:02:19.837570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.833309    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td" event={"ID":"2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b","Type":"ContainerDied","Data":"e04b3d7f8736a7890a948a868170c54ef906a7e370be57d5e8e8686f4962b506"}
Nov 01 09:02:19.844251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.844140    4314 generic.go:356] "Generic (PLEG): container finished" podID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerID="342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61" exitCode=0
Nov 01 09:02:19.844251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.844176    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" event={"ID":"4636a672-4b3c-4f31-b8b6-3eef2dde47db","Type":"ContainerDied","Data":"342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61"}
Nov 01 09:02:19.844251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.844225    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"
Nov 01 09:02:19.844251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.844250    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq" event={"ID":"4636a672-4b3c-4f31-b8b6-3eef2dde47db","Type":"ContainerDied","Data":"dc80d8f7423bb8a18a3c7e912a93137e9f3ba411904e3125c5eb4b6637bffbb9"}
Nov 01 09:02:19.847066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.847050    4314 scope.go:122] "RemoveContainer" containerID="9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0"
Nov 01 09:02:19.848673 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:19.848649    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0\": container with ID starting with 9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0 not found: ID does not exist" containerID="9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0"
Nov 01 09:02:19.848780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.848684    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0"} err="failed to get container status \"9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0\": rpc error: code = NotFound desc = could not find container \"9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0\": container with ID starting with 9d7de2f716e95cb34fcbe6772803f448a3582038289fa7e464dc6b6fcd4e97b0 not found: ID does not exist"
Nov 01 09:02:19.848780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.848707    4314 scope.go:122] "RemoveContainer" containerID="33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263"
Nov 01 09:02:19.856768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856745    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856769    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856781    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856789    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856798    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856809    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd487\" (UniqueName: \"kubernetes.io/projected/119bbcf8-3604-44d5-918c-7b5c5f10aabd-kube-api-access-nd487\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856817    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856835    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856843    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856853    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bbcf8-3604-44d5-918c-7b5c5f10aabd-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856862    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856871    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4636a672-4b3c-4f31-b8b6-3eef2dde47db-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856883    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf477d9c-9a2f-4618-998a-150e6baa9147-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856896    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqc9d\" (UniqueName: \"kubernetes.io/projected/bf477d9c-9a2f-4618-998a-150e6baa9147-kube-api-access-pqc9d\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.856911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.856908    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4636a672-4b3c-4f31-b8b6-3eef2dde47db-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:19.861009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.860988    4314 scope.go:122] "RemoveContainer" containerID="33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263"
Nov 01 09:02:19.861342 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:19.861317    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263\": container with ID starting with 33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263 not found: ID does not exist" containerID="33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263"
Nov 01 09:02:19.861399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.861351    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263"} err="failed to get container status \"33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263\": rpc error: code = NotFound desc = could not find container \"33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263\": container with ID starting with 33f6dd5c00e58cc3fb20392a98fe72f42c3e6d4465ce2495566413c30a5ad263 not found: ID does not exist"
Nov 01 09:02:19.861399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.861372    4314 scope.go:122] "RemoveContainer" containerID="04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a"
Nov 01 09:02:19.871059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.871041    4314 scope.go:122] "RemoveContainer" containerID="04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a"
Nov 01 09:02:19.871352 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:19.871332    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a\": container with ID starting with 04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a not found: ID does not exist" containerID="04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a"
Nov 01 09:02:19.871403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.871363    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a"} err="failed to get container status \"04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a\": rpc error: code = NotFound desc = could not find container \"04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a\": container with ID starting with 04994196d86a0a9fcd465c0523490d763fcceaeac9b551f83845c7a9f2b4cc7a not found: ID does not exist"
Nov 01 09:02:19.871403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.871381    4314 scope.go:122] "RemoveContainer" containerID="342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61"
Nov 01 09:02:19.883588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.883562    4314 scope.go:122] "RemoveContainer" containerID="342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61"
Nov 01 09:02:19.883973 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:19.883951    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61\": container with ID starting with 342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61 not found: ID does not exist" containerID="342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61"
Nov 01 09:02:19.884043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:19.883981    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61"} err="failed to get container status \"342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61\": rpc error: code = NotFound desc = could not find container \"342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61\": container with ID starting with 342af162a8d4e7e488dcefe32f7cb03a29f8727319e40b49c1a827f97e207d61 not found: ID does not exist"
Nov 01 09:02:20.220452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.220415    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"]
Nov 01 09:02:20.224546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.224509    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5n9td"]
Nov 01 09:02:20.345716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.345683    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"]
Nov 01 09:02:20.361808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.361778    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-sdwlj"]
Nov 01 09:02:20.462905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.462867    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"]
Nov 01 09:02:20.467805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.467777    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-lnjmm"]
Nov 01 09:02:20.568983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.568952    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"]
Nov 01 09:02:20.574013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.573988    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tfphq"]
Nov 01 09:02:20.929437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.929362    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" path="/var/lib/kubelet/pods/119bbcf8-3604-44d5-918c-7b5c5f10aabd/volumes"
Nov 01 09:02:20.929906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.929889    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" path="/var/lib/kubelet/pods/2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b/volumes"
Nov 01 09:02:20.930310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.930297    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" path="/var/lib/kubelet/pods/4636a672-4b3c-4f31-b8b6-3eef2dde47db/volumes"
Nov 01 09:02:20.931003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.930988    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1bbc18c-3927-4686-9faf-5147d4ccb398" path="/var/lib/kubelet/pods/b1bbc18c-3927-4686-9faf-5147d4ccb398/volumes"
Nov 01 09:02:20.931394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:20.931382    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" path="/var/lib/kubelet/pods/bf477d9c-9a2f-4618-998a-150e6baa9147/volumes"
Nov 01 09:02:21.197861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:21.197781    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:21.489387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:21.489357    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:21.634552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:21.634504    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:22.546765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:22.546730    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:23.395711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:23.395676    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:23.771228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:23.771193    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:23.916575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:23.916524    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:25.664137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:25.664094    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:27.397303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:27.397254    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:27.564009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:27.563978    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:27.706750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:27.706657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:29.794913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:29.794862    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:32.906307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:32.906266    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:38.028135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:38.028102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:41.337266 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:41.337218    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:02:41.370925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.370893    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"]
Nov 01 09:02:41.371528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371506    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b1bbc18c-3927-4686-9faf-5147d4ccb398" containerName="payload-processing"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371543    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1bbc18c-3927-4686-9faf-5147d4ccb398" containerName="payload-processing"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371583    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371591    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371617    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371625    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371635    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371642    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy"
Nov 01 09:02:41.371658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371657    4314 cpu_manager.go:414] "containerMap: removing container" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy"
Nov 01 09:02:41.372055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371664    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy"
Nov 01 09:02:41.372055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371814    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4636a672-4b3c-4f31-b8b6-3eef2dde47db" containerName="istio-proxy"
Nov 01 09:02:41.372055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371829    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf477d9c-9a2f-4618-998a-150e6baa9147" containerName="istio-proxy"
Nov 01 09:02:41.372055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371841    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="119bbcf8-3604-44d5-918c-7b5c5f10aabd" containerName="istio-proxy"
Nov 01 09:02:41.372055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371854    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1bbc18c-3927-4686-9faf-5147d4ccb398" containerName="payload-processing"
Nov 01 09:02:41.372055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.371867    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7fcfe5-6a1d-4ff3-8634-7c151b155a5b" containerName="istio-proxy"
Nov 01 09:02:41.375494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.375473    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.424175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.424142    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d31acd45-openshift-default-dockercfg-p6mlc\""
Nov 01 09:02:41.459864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.459832    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.459864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.459865    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460013    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460083    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460205    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460239    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj69g\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460333    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460391    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.460447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.460426    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.474252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.474221    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"]
Nov 01 09:02:41.561855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.561816    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.561869    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.561962    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.561992    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj69g\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562028    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562062    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562109    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562211    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562241    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562365    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562586    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562673    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.562873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562843    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.563010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.562956    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.564465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.564445    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.564604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.564584    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.619607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.619548    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.619736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.619681    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj69g\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.689083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.689056    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:41.860063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:41.860029    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:42.105893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:42.105863    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"]
Nov 01 09:02:42.107490 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:02:42.107463    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1389b0ef_8fa3_4e72_ae50_2ce050fb2899.slice/crio-b312ffd728d520c4377c9c25bdfb3433ad6ac8288c655248da740ceeffa8540f WatchSource:0}: Error finding container b312ffd728d520c4377c9c25bdfb3433ad6ac8288c655248da740ceeffa8540f: Status 404 returned error can't find the container with id b312ffd728d520c4377c9c25bdfb3433ad6ac8288c655248da740ceeffa8540f
Nov 01 09:02:42.110099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:42.110063    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:02:42.110206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:42.110146    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:02:42.110206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:42.110191    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:02:42.487554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:42.487471    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:42.987299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:42.987255    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:43.021686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:43.021651    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" event={"ID":"1389b0ef-8fa3-4e72-ae50-2ce050fb2899","Type":"ContainerStarted","Data":"f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491"}
Nov 01 09:02:43.021994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:43.021712    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" event={"ID":"1389b0ef-8fa3-4e72-ae50-2ce050fb2899","Type":"ContainerStarted","Data":"b312ffd728d520c4377c9c25bdfb3433ad6ac8288c655248da740ceeffa8540f"}
Nov 01 09:02:43.230470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:43.230402    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podStartSLOduration=2.230383121 podStartE2EDuration="2.230383121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:02:43.181131902 +0000 UTC m=+1836.375526454" watchObservedRunningTime="2026-11-01 09:02:43.230383121 +0000 UTC m=+1836.424777676"
Nov 01 09:02:43.690072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:43.690035    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:43.691985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:43.691957    4314 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 09:02:43.692128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:43.692006    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 09:02:44.504837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:44.504802    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:44.690339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:44.690287    4314 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 09:02:44.690901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:44.690351    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 09:02:45.689647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:45.689592    4314 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 09:02:45.689843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:45.689660    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 09:02:46.689888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:46.689847    4314 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 09:02:46.690491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:46.690455    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 09:02:47.690042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:47.689998    4314 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 09:02:47.690561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:47.690070    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 09:02:48.689932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:48.689848    4314 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 09:02:48.689932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:48.689910    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 09:02:48.856232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:48.856191    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:49.621902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:49.621870    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:49.694070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:49.694039    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:49.694517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:49.694496    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:49.695344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:49.695320    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:02:50.171848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:50.171812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:50.314873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:50.314835    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:51.587298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:51.587259    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:51.731030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:51.730992    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:53.078695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:53.078659    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:53.563293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:53.563259    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"]
Nov 01 09:02:53.563493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:53.563471    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" containerID="cri-o://fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76" gracePeriod=30
Nov 01 09:02:53.566863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:53.566839    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:54.333413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:54.333379    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:54.670229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:54.670155    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:54.813001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:54.812966    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:02:55.815098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.815065    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"]
Nov 01 09:02:55.819363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.819347    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.868005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.867970    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-991c28-openshift-default-dockercfg-tvzzv\""
Nov 01 09:02:55.917749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.917719    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"]
Nov 01 09:02:55.930612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930583    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930651    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930710    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930746    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930789    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930829    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930868    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930932    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwn8\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:55.930996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:55.930981    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.031688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031659    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.031862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031697    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.031862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031726    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.031862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031766    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.031862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031802    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.031862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031858    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwn8\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.031902    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032007    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032081    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032111    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032168    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032473    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032464    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.032812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.032789    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.034412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.034380    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.034686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.034666    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.088955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.088892    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.089068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.088983    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwn8\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.133524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.133485    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:56.550548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.550505    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"]
Nov 01 09:02:56.550699 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:02:56.550680    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6ac689_a6a8_4ae6_a1eb_c7055b0b1d28.slice/crio-98016881660ec11151c8d185654e1b62c7eb8ba34116ce11023495c143364a65 WatchSource:0}: Error finding container 98016881660ec11151c8d185654e1b62c7eb8ba34116ce11023495c143364a65: Status 404 returned error can't find the container with id 98016881660ec11151c8d185654e1b62c7eb8ba34116ce11023495c143364a65
Nov 01 09:02:56.553109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.553073    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:02:56.553192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.553142    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:02:56.553236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:56.553189    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:02:57.115296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.115246    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" event={"ID":"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28","Type":"ContainerStarted","Data":"08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f"}
Nov 01 09:02:57.115736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.115318    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" event={"ID":"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28","Type":"ContainerStarted","Data":"98016881660ec11151c8d185654e1b62c7eb8ba34116ce11023495c143364a65"}
Nov 01 09:02:57.133610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.133583    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:02:57.135147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.135116    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 01 09:02:57.135290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.135167    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 01 09:02:57.232673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.232624    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podStartSLOduration=2.232606432 podStartE2EDuration="2.232606432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:02:57.23129659 +0000 UTC m=+1850.425691145" watchObservedRunningTime="2026-11-01 09:02:57.232606432 +0000 UTC m=+1850.427000986"
Nov 01 09:02:57.806615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.806575    4314 patch_prober.go:29] interesting pod/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:02:57.806788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:57.806639    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:02:58.134553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.134451    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 01 09:02:58.134553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.134506    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 01 09:02:58.792830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.792803    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 09:02:58.897793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.897701    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.897793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.897754    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.897793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.897791    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.897832    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.897998    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898030    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898070    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898121    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898127    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:58.898292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898162    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:58.898292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898173    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data\") pod \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\" (UID: \"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d\") "
Nov 01 09:02:58.898474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898366    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:58.898551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898464    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:58.898669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898640    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:58.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898861    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:58.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898890    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:58.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898901    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:58.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898915    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:58.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.898928    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:58.900274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.900235    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "kube-api-access-ljpbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:58.900361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.900291    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:58.900483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.900461    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:02:58.900569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.900523    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy" pod "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" (UID: "dca0a0aa-ffe1-4ac5-b35d-326f3830df8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:58.999998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.999963    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljpbn\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-kube-api-access-ljpbn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:58.999998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:58.999999    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:59.000201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.000013    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:59.000201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.000027    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:59.131022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.130990    4314 generic.go:356] "Generic (PLEG): container finished" podID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerID="fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76" exitCode=0
Nov 01 09:02:59.131204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.131049    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"
Nov 01 09:02:59.131204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.131080    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" event={"ID":"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d","Type":"ContainerDied","Data":"fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76"}
Nov 01 09:02:59.131204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.131143    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn" event={"ID":"dca0a0aa-ffe1-4ac5-b35d-326f3830df8d","Type":"ContainerDied","Data":"d65193bd5942a987ed608554ee9e583df5fbb5ca1a75b918b14caee1b5381838"}
Nov 01 09:02:59.131204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.131168    4314 scope.go:122] "RemoveContainer" containerID="fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76"
Nov 01 09:02:59.133840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.133802    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 01 09:02:59.133965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.133869    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 01 09:02:59.141822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.141803    4314 scope.go:122] "RemoveContainer" containerID="7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"
Nov 01 09:02:59.150712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.150694    4314 scope.go:122] "RemoveContainer" containerID="fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76"
Nov 01 09:02:59.151005 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:59.150986    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76\": container with ID starting with fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76 not found: ID does not exist" containerID="fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76"
Nov 01 09:02:59.151066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.151012    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76"} err="failed to get container status \"fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76\": rpc error: code = NotFound desc = could not find container \"fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76\": container with ID starting with fbe4a859c436db1700451d605ac821aa9bc4fd7b5454cdb2935d84773353ad76 not found: ID does not exist"
Nov 01 09:02:59.151066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.151031    4314 scope.go:122] "RemoveContainer" containerID="7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"
Nov 01 09:02:59.151258 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:02:59.151237    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe\": container with ID starting with 7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe not found: ID does not exist" containerID="7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"
Nov 01 09:02:59.151315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.151269    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe"} err="failed to get container status \"7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe\": rpc error: code = NotFound desc = could not find container \"7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe\": container with ID starting with 7fb4e9db98a02b75147a2ef469d82052db62eac0a546cc8ea94b100dfa2f86fe not found: ID does not exist"
Nov 01 09:02:59.292668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.292624    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"]
Nov 01 09:02:59.295858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.295834    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-03832c-openshift-default-5ff5c8d467-8zbfn"]
Nov 01 09:02:59.886372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:02:59.886334    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:00.134732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:00.134700    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 01 09:03:00.134917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:00.134755    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 01 09:03:00.809327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:00.809296    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:00.929522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:00.929492    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" path="/var/lib/kubelet/pods/dca0a0aa-ffe1-4ac5-b35d-326f3830df8d/volumes"
Nov 01 09:03:01.134541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:01.134445    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 01 09:03:01.134541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:01.134511    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 01 09:03:01.381060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:01.381024    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:01.525243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:01.525204    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:02.048904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:02.048865    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"]
Nov 01 09:03:02.095033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:02.094998    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2af39e-cxx2q"]
Nov 01 09:03:02.134212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:02.134184    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 01 09:03:02.134355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:02.134233    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 01 09:03:02.929278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:02.929248    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef04bc2d-eb57-44be-80e6-26f98467f4a6" path="/var/lib/kubelet/pods/ef04bc2d-eb57-44be-80e6-26f98467f4a6/volumes"
Nov 01 09:03:03.139172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:03.139144    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:03:03.139603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:03.139416    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:03:03.140169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:03.140149    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:03:05.147872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:05.147834    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:06.379191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:06.379143    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:07.977082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:07.977049    4314 scope.go:122] "RemoveContainer" containerID="23d8bb97217c201fbe2fd9e02db226abd522d9a9fcafbc50418d1fd77cc40f71"
Nov 01 09:03:10.383068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:10.383037    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:10.528737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:10.528705    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:11.323063 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:11.323025    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:03:14.997879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:14.997840    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:15.987598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:15.987564    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:16.136677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:16.136642    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:16.285008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:16.284918    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:17.075201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:17.075156    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:19.522855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:19.522821    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:19.668398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:19.668367    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:21.021662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:21.021630    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:21.537185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:21.537148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:21.544403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:21.544366    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"]
Nov 01 09:03:21.544745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:21.544715    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy" containerID="cri-o://6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59" gracePeriod=30
Nov 01 09:03:22.310485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:22.310452    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:23.467022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:23.466980    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:23.610665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:23.610599    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:23.957989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:23.957958    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"]
Nov 01 09:03:23.958173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:23.958151    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy" containerID="cri-o://f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491" gracePeriod=30
Nov 01 09:03:23.967240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:23.967212    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:24.427484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:24.427452    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:24.574277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:24.574242    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:26.804655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.804629    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 09:03:26.878463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878379    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878430    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878464    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878602    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878628    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878725    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878755    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:26.878903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878787    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.878903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878847    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:26.878903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878869    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.879207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.878981    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert\") pod \"0673bae6-a048-4b51-8444-988db5035dec\" (UID: \"0673bae6-a048-4b51-8444-988db5035dec\") "
Nov 01 09:03:26.879207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879042    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:26.879207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879067    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:26.879454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879426    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:03:26.879641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879596    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.879641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879621    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.879641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879637    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.879829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879652    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.879829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.879665    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bae6-a048-4b51-8444-988db5035dec-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.881067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.881044    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:26.881204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.881166    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:03:26.881317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.881244    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:03:26.881383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.881339    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj" pod "0673bae6-a048-4b51-8444-988db5035dec" (UID: "0673bae6-a048-4b51-8444-988db5035dec"). InnerVolumeSpecName "kube-api-access-cnvjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:03:26.981299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.981264    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bae6-a048-4b51-8444-988db5035dec-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.981299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.981302    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bae6-a048-4b51-8444-988db5035dec-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.981484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.981316    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:26.981484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:26.981330    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnvjj\" (UniqueName: \"kubernetes.io/projected/0673bae6-a048-4b51-8444-988db5035dec-kube-api-access-cnvjj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:27.306898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.306860    4314 generic.go:356] "Generic (PLEG): container finished" podID="0673bae6-a048-4b51-8444-988db5035dec" containerID="6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59" exitCode=0
Nov 01 09:03:27.307080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.306914    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"
Nov 01 09:03:27.307080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.306943    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" event={"ID":"0673bae6-a048-4b51-8444-988db5035dec","Type":"ContainerDied","Data":"6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59"}
Nov 01 09:03:27.307080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.307002    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n" event={"ID":"0673bae6-a048-4b51-8444-988db5035dec","Type":"ContainerDied","Data":"8b3825b2fe2e8d10fb9b0c17939f3c12916060d44f17ad23504e2581827f530e"}
Nov 01 09:03:27.307080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.307026    4314 scope.go:122] "RemoveContainer" containerID="6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59"
Nov 01 09:03:27.316759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.316743    4314 scope.go:122] "RemoveContainer" containerID="3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"
Nov 01 09:03:27.325236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.325221    4314 scope.go:122] "RemoveContainer" containerID="6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59"
Nov 01 09:03:27.325502 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:27.325483    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59\": container with ID starting with 6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59 not found: ID does not exist" containerID="6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59"
Nov 01 09:03:27.325645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.325515    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59"} err="failed to get container status \"6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59\": rpc error: code = NotFound desc = could not find container \"6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59\": container with ID starting with 6a95007b0d17cf42b09bf57547290e3d89ab6c5be5d2df0fe00432616fc30e59 not found: ID does not exist"
Nov 01 09:03:27.325645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.325615    4314 scope.go:122] "RemoveContainer" containerID="3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"
Nov 01 09:03:27.325936 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:27.325910    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1\": container with ID starting with 3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1 not found: ID does not exist" containerID="3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"
Nov 01 09:03:27.325984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.325946    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1"} err="failed to get container status \"3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1\": rpc error: code = NotFound desc = could not find container \"3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1\": container with ID starting with 3c3d37f8c2751b8e1dbef6ba416c0cf3f4b2923d5846f0baf32e1c5b0e8a30a1 not found: ID does not exist"
Nov 01 09:03:27.461757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.461720    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"]
Nov 01 09:03:27.507753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:27.507714    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3901b8-openshift-default-5fd8c96bf5-gqx5n"]
Nov 01 09:03:28.343802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:28.343765    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:28.929244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:28.929214    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0673bae6-a048-4b51-8444-988db5035dec" path="/var/lib/kubelet/pods/0673bae6-a048-4b51-8444-988db5035dec/volumes"
Nov 01 09:03:28.987252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:28.987225    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:29.213036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.213012    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:03:29.312959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.312922    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.312969    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313011    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313086    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313144    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313214    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313242    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313272    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:29.313371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313343    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313349    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:29.313685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313389    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo\") pod \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\" (UID: \"1389b0ef-8fa3-4e72-ae50-2ce050fb2899\") "
Nov 01 09:03:29.313685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313497    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:29.313685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313584    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:03:29.313685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313616    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:29.313989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313968    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.314073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.313993    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.314073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.314008    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.314073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.314022    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.314073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.314035    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.315465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.315436    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:03:29.315465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.315450    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "kube-api-access-vj69g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:03:29.315632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.315519    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:03:29.315632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.315557    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token" pod "1389b0ef-8fa3-4e72-ae50-2ce050fb2899" (UID: "1389b0ef-8fa3-4e72-ae50-2ce050fb2899"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:03:29.325707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.324657    4314 generic.go:356] "Generic (PLEG): container finished" podID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerID="f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491" exitCode=0
Nov 01 09:03:29.325707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.324698    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" event={"ID":"1389b0ef-8fa3-4e72-ae50-2ce050fb2899","Type":"ContainerDied","Data":"f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491"}
Nov 01 09:03:29.325707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.324718    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"
Nov 01 09:03:29.325707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.324769    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j" event={"ID":"1389b0ef-8fa3-4e72-ae50-2ce050fb2899","Type":"ContainerDied","Data":"b312ffd728d520c4377c9c25bdfb3433ad6ac8288c655248da740ceeffa8540f"}
Nov 01 09:03:29.325707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.324803    4314 scope.go:122] "RemoveContainer" containerID="f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491"
Nov 01 09:03:29.341790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.341773    4314 scope.go:122] "RemoveContainer" containerID="f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491"
Nov 01 09:03:29.342076 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:29.342056    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491\": container with ID starting with f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491 not found: ID does not exist" containerID="f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491"
Nov 01 09:03:29.342134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.342084    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491"} err="failed to get container status \"f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491\": rpc error: code = NotFound desc = could not find container \"f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491\": container with ID starting with f920f932e71ffe4d1ef36a1c87c502d331b8e29b67353bc84d803c7650b45491 not found: ID does not exist"
Nov 01 09:03:29.415261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.415220    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.415261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.415249    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.415261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.415261    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj69g\" (UniqueName: \"kubernetes.io/projected/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-kube-api-access-vj69g\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.415261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.415270    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1389b0ef-8fa3-4e72-ae50-2ce050fb2899-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:29.481356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.481321    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"]
Nov 01 09:03:29.485473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.485444    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-2q68j"]
Nov 01 09:03:29.615434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.615394    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:29.782229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:29.782148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:30.929202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:30.929168    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" path="/var/lib/kubelet/pods/1389b0ef-8fa3-4e72-ae50-2ce050fb2899/volumes"
Nov 01 09:03:32.178826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:32.178790    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:32.361124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:32.361092    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:32.514373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:32.514339    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:33.044906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.044876    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:33.612423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.612389    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"]
Nov 01 09:03:33.612942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.612929    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy"
Nov 01 09:03:33.612984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.612943    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy"
Nov 01 09:03:33.613020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.612986    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy"
Nov 01 09:03:33.613020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.612992    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy"
Nov 01 09:03:33.613020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613002    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy"
Nov 01 09:03:33.613020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613007    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy"
Nov 01 09:03:33.613020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613017    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy"
Nov 01 09:03:33.613020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613022    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy"
Nov 01 09:03:33.613207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613102    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy"
Nov 01 09:03:33.613207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613111    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy"
Nov 01 09:03:33.613207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613118    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy"
Nov 01 09:03:33.613207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.613124    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1389b0ef-8fa3-4e72-ae50-2ce050fb2899" containerName="istio-proxy"
Nov 01 09:03:33.617013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.616987    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.665701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.665667    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-23fa20-openshift-default-dockercfg-thzjx\""
Nov 01 09:03:33.669484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669441    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669498    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669583    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669700    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669772    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669808    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669844    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhpv6\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669892    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.669989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.669935    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.703661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.703626    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:33.716824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.716795    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"]
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771304    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771372    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771409    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhpv6\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771455    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771496    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771546    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771589    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771666    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.771804    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.772596    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.774591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.772872    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.824239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.801937    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.824239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.822768    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.824634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.824608    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.854557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.844144    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.854557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.846239    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.857231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.856436    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhpv6\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.858695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.858671    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert\") pod \"e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") " pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.931221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.931142    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:33.940144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:33.940108    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:34.249718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.249681    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:34.305416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.304983    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"]
Nov 01 09:03:34.352980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.352948    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"]
Nov 01 09:03:34.354105 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:03:34.354077    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d60b6bf_33f2_4162_a2d7_13ace4e85faa.slice/crio-66ee66d174ac195d6c45cd50948e094e4a26fb759a723d27aea0ad4dd46ffd12 WatchSource:0}: Error finding container 66ee66d174ac195d6c45cd50948e094e4a26fb759a723d27aea0ad4dd46ffd12: Status 404 returned error can't find the container with id 66ee66d174ac195d6c45cd50948e094e4a26fb759a723d27aea0ad4dd46ffd12
Nov 01 09:03:34.356526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.356496    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:03:34.356622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.356580    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:03:34.356622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.356614    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:03:34.372974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.372952    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" event={"ID":"9d60b6bf-33f2-4162-a2d7-13ace4e85faa","Type":"ContainerStarted","Data":"66ee66d174ac195d6c45cd50948e094e4a26fb759a723d27aea0ad4dd46ffd12"}
Nov 01 09:03:34.397989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.397963    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"]
Nov 01 09:03:34.402659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.402613    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:34.647129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.647049    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:34.800257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:34.799993    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:35.353275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:35.353228    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:35.380976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:35.380936    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" event={"ID":"9d60b6bf-33f2-4162-a2d7-13ace4e85faa","Type":"ContainerStarted","Data":"a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"}
Nov 01 09:03:35.495720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:35.495664    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podStartSLOduration=2.495646097 podStartE2EDuration="2.495646097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:03:35.493075146 +0000 UTC m=+1888.687469701" watchObservedRunningTime="2026-11-01 09:03:35.495646097 +0000 UTC m=+1888.690040652"
Nov 01 09:03:35.675609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:35.672195    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:35.932325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:35.932240    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:35.937355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:35.937332    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:36.386605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:36.386574    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:36.387520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:36.387498    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:03:39.306363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.306323    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h" podUID="1eb9c8e5-9634-4020-8e3f-febedab10fa1" containerName="payload-processing" containerID="cri-o://61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908" gracePeriod=25
Nov 01 09:03:39.398648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.398588    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd" podUID="28402fdc-4c72-4a12-ad1a-04c567a29a0e" containerName="payload-pre-processing" containerID="cri-o://318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414" gracePeriod=25
Nov 01 09:03:39.437782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.437753    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:39.588817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.588725    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:39.674746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.671469    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:03:39.712500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.712472    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:03:39.742478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.742447    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:39.749863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.749841    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume\") pod \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") "
Nov 01 09:03:39.750041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.750026    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7\" (UniqueName: \"kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7\") pod \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\" (UID: \"1eb9c8e5-9634-4020-8e3f-febedab10fa1\") "
Nov 01 09:03:39.750197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.750171    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume" pod "1eb9c8e5-9634-4020-8e3f-febedab10fa1" (UID: "1eb9c8e5-9634-4020-8e3f-febedab10fa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:03:39.750497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.750478    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb9c8e5-9634-4020-8e3f-febedab10fa1-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:39.752094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.752074    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7" pod "1eb9c8e5-9634-4020-8e3f-febedab10fa1" (UID: "1eb9c8e5-9634-4020-8e3f-febedab10fa1"). InnerVolumeSpecName "kube-api-access-n6wg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:03:39.851340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.851268    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume\") pod \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") "
Nov 01 09:03:39.851488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.851410    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f\" (UniqueName: \"kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f\") pod \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\" (UID: \"28402fdc-4c72-4a12-ad1a-04c567a29a0e\") "
Nov 01 09:03:39.851710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.851682    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume" pod "28402fdc-4c72-4a12-ad1a-04c567a29a0e" (UID: "28402fdc-4c72-4a12-ad1a-04c567a29a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:03:39.851883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.851866    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wg7\" (UniqueName: \"kubernetes.io/projected/1eb9c8e5-9634-4020-8e3f-febedab10fa1-kube-api-access-n6wg7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:39.851946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.851887    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28402fdc-4c72-4a12-ad1a-04c567a29a0e-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:39.853603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.853580    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f" pod "28402fdc-4c72-4a12-ad1a-04c567a29a0e" (UID: "28402fdc-4c72-4a12-ad1a-04c567a29a0e"). InnerVolumeSpecName "kube-api-access-nd68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:03:39.899469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.899435    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:39.952748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:39.952718    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd68f\" (UniqueName: \"kubernetes.io/projected/28402fdc-4c72-4a12-ad1a-04c567a29a0e-kube-api-access-nd68f\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:03:40.421066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.421033    4314 generic.go:356] "Generic (PLEG): container finished" podID="28402fdc-4c72-4a12-ad1a-04c567a29a0e" containerID="318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414" exitCode=0
Nov 01 09:03:40.421477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.421078    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"
Nov 01 09:03:40.421477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.421105    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd" event={"ID":"28402fdc-4c72-4a12-ad1a-04c567a29a0e","Type":"ContainerDied","Data":"318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414"}
Nov 01 09:03:40.421477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.421153    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd" event={"ID":"28402fdc-4c72-4a12-ad1a-04c567a29a0e","Type":"ContainerDied","Data":"608aecfe900642050a4681a74bd63df09a7663ec2cd4ec37907dd8d8fceca5a1"}
Nov 01 09:03:40.421477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.421180    4314 scope.go:122] "RemoveContainer" containerID="318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414"
Nov 01 09:03:40.422567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.422546    4314 generic.go:356] "Generic (PLEG): container finished" podID="1eb9c8e5-9634-4020-8e3f-febedab10fa1" containerID="61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908" exitCode=0
Nov 01 09:03:40.422640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.422582    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"
Nov 01 09:03:40.422692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.422634    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h" event={"ID":"1eb9c8e5-9634-4020-8e3f-febedab10fa1","Type":"ContainerDied","Data":"61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908"}
Nov 01 09:03:40.422750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.422691    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h" event={"ID":"1eb9c8e5-9634-4020-8e3f-febedab10fa1","Type":"ContainerDied","Data":"bb44a2607f97b0193969b1d83763536887b3af76a9264af278ad9d50a3262bfc"}
Nov 01 09:03:40.431654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.431638    4314 scope.go:122] "RemoveContainer" containerID="318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414"
Nov 01 09:03:40.432055 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:40.431924    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414\": container with ID starting with 318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414 not found: ID does not exist" containerID="318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414"
Nov 01 09:03:40.432055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.431949    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414"} err="failed to get container status \"318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414\": rpc error: code = NotFound desc = could not find container \"318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414\": container with ID starting with 318709418e38434401e28c3f012393b0025a4ecaf5a7c06f9efdae52b5f91414 not found: ID does not exist"
Nov 01 09:03:40.432055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.431967    4314 scope.go:122] "RemoveContainer" containerID="61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908"
Nov 01 09:03:40.441052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.441034    4314 scope.go:122] "RemoveContainer" containerID="61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908"
Nov 01 09:03:40.441342 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:40.441324    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908\": container with ID starting with 61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908 not found: ID does not exist" containerID="61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908"
Nov 01 09:03:40.441399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.441352    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908"} err="failed to get container status \"61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908\": rpc error: code = NotFound desc = could not find container \"61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908\": container with ID starting with 61f2d863952bc3d85c656a835245004c4552aef1ca33b47b8c9d854aef991908 not found: ID does not exist"
Nov 01 09:03:40.581745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.581714    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"]
Nov 01 09:03:40.627719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.627683    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-dcc4d5b89-7654h"]
Nov 01 09:03:40.687526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.687441    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"]
Nov 01 09:03:40.691789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.691763    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-6457b74559-bdgcd"]
Nov 01 09:03:40.930072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.930041    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb9c8e5-9634-4020-8e3f-febedab10fa1" path="/var/lib/kubelet/pods/1eb9c8e5-9634-4020-8e3f-febedab10fa1/volumes"
Nov 01 09:03:40.930427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:40.930414    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28402fdc-4c72-4a12-ad1a-04c567a29a0e" path="/var/lib/kubelet/pods/28402fdc-4c72-4a12-ad1a-04c567a29a0e/volumes"
Nov 01 09:03:41.049680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:41.049624    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:41.322331 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:03:41.322252    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:03:43.266354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.266309    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:43.425340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.425303    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:43.664950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.664867    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"]
Nov 01 09:03:43.665411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665396    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1eb9c8e5-9634-4020-8e3f-febedab10fa1" containerName="payload-processing"
Nov 01 09:03:43.665450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665412    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb9c8e5-9634-4020-8e3f-febedab10fa1" containerName="payload-processing"
Nov 01 09:03:43.665450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665445    4314 cpu_manager.go:414] "containerMap: removing container" podUID="28402fdc-4c72-4a12-ad1a-04c567a29a0e" containerName="payload-pre-processing"
Nov 01 09:03:43.665522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665450    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28402fdc-4c72-4a12-ad1a-04c567a29a0e" containerName="payload-pre-processing"
Nov 01 09:03:43.665522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665464    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy"
Nov 01 09:03:43.665522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665469    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca0a0aa-ffe1-4ac5-b35d-326f3830df8d" containerName="istio-proxy"
Nov 01 09:03:43.665643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665574    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="28402fdc-4c72-4a12-ad1a-04c567a29a0e" containerName="payload-pre-processing"
Nov 01 09:03:43.665643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665584    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0673bae6-a048-4b51-8444-988db5035dec" containerName="istio-proxy"
Nov 01 09:03:43.665643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.665592    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb9c8e5-9634-4020-8e3f-febedab10fa1" containerName="payload-processing"
Nov 01 09:03:43.670256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.670239    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.722570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.722524    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"]
Nov 01 09:03:43.764923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.764891    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-23fa20-dockercfg-pmw9j\""
Nov 01 09:03:43.764923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.764746    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-23fa20\""
Nov 01 09:03:43.808556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.804228    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdcf\" (UniqueName: \"kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.808556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.804425    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.835418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.835378    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"]
Nov 01 09:03:43.840369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.840333    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:43.893624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.893592    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"]
Nov 01 09:03:43.905707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.905666    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcwk\" (UniqueName: \"kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk\") pod \"payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:43.905880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.905826    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcf\" (UniqueName: \"kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.906033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.905973    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:43.906033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.906005    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.906956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.906934    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.927088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.927004    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:43.963197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.963171    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdcf\" (UniqueName: \"kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:43.983731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:43.983700    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:44.007409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.007375    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:44.007544 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.007515    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcwk\" (UniqueName: \"kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk\") pod \"payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:44.008260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.008238    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:44.086928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.079347    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcwk\" (UniqueName: \"kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk\") pod \"payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:44.165828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.165791    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:44.371055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.371024    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"]
Nov 01 09:03:44.371669 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:03:44.371643    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a898fed_a7df_4614_88b5_62056f21ed9a.slice/crio-983a15293afbc3f391140d23d11ca9c3932ba1c724da11080448d210395c3761 WatchSource:0}: Error finding container 983a15293afbc3f391140d23d11ca9c3932ba1c724da11080448d210395c3761: Status 404 returned error can't find the container with id 983a15293afbc3f391140d23d11ca9c3932ba1c724da11080448d210395c3761
Nov 01 09:03:44.452052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.452021    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt" event={"ID":"4a898fed-a7df-4614-88b5-62056f21ed9a","Type":"ContainerStarted","Data":"983a15293afbc3f391140d23d11ca9c3932ba1c724da11080448d210395c3761"}
Nov 01 09:03:44.563484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:44.563456    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"]
Nov 01 09:03:44.564878 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:03:44.564848    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b08a1b_634d_4d56_bfa9_83bcce0de3e4.slice/crio-a86a74413abac407dd034c4d8ce1f3ca9e6db89dfee670cf3d1d6af1c22c8129 WatchSource:0}: Error finding container a86a74413abac407dd034c4d8ce1f3ca9e6db89dfee670cf3d1d6af1c22c8129: Status 404 returned error can't find the container with id a86a74413abac407dd034c4d8ce1f3ca9e6db89dfee670cf3d1d6af1c22c8129
Nov 01 09:03:45.459735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.459687    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt" event={"ID":"4a898fed-a7df-4614-88b5-62056f21ed9a","Type":"ContainerStarted","Data":"db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2"}
Nov 01 09:03:45.460163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.459799    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:45.461592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.461562    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67" event={"ID":"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4","Type":"ContainerStarted","Data":"442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b"}
Nov 01 09:03:45.461705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.461608    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67" event={"ID":"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4","Type":"ContainerStarted","Data":"a86a74413abac407dd034c4d8ce1f3ca9e6db89dfee670cf3d1d6af1c22c8129"}
Nov 01 09:03:45.461705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.461672    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:45.568971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.568912    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt" podStartSLOduration=2.568898533 podStartE2EDuration="2.568898533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:03:45.567829203 +0000 UTC m=+1898.762223756" watchObservedRunningTime="2026-11-01 09:03:45.568898533 +0000 UTC m=+1898.763293133"
Nov 01 09:03:45.678709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.678652    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67" podStartSLOduration=2.678637523 podStartE2EDuration="2.678637523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:03:45.676004699 +0000 UTC m=+1898.870399251" watchObservedRunningTime="2026-11-01 09:03:45.678637523 +0000 UTC m=+1898.873032253"
Nov 01 09:03:45.695977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:45.695943    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:49.594309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:49.594243    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:49.743311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:49.743265    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:50.045869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:50.045822    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"]
Nov 01 09:03:50.091764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:50.091724    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a28168-7fwxg"]
Nov 01 09:03:50.845495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:50.845459    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:50.930214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:50.930180    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9109b660-10be-47f2-a6f6-d39ebc028aed" path="/var/lib/kubelet/pods/9109b660-10be-47f2-a6f6-d39ebc028aed/volumes"
Nov 01 09:03:51.201692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:51.201596    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:51.673763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:51.673725    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:52.661091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:52.661047    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:53.315554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:53.305969    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:53.808462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:53.808388    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:54.752837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:54.752800    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:55.249402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:55.249351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:56.339603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.339570    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"]
Nov 01 09:03:56.345139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.345109    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.395453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.395418    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73660a\""
Nov 01 09:03:56.428765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.428724    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.428964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.428783    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdcz\" (UniqueName: \"kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz\") pod \"payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.441603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.441563    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73660a-dockercfg-hc2cb\""
Nov 01 09:03:56.444858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.444827    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"]
Nov 01 09:03:56.470375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.470342    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:03:56.470586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.470415    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:03:56.530079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.530041    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.530303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.530094    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdcz\" (UniqueName: \"kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz\") pod \"payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.530802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.530777    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.587861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.587828    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdcz\" (UniqueName: \"kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz\") pod \"payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.661452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.661358    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:03:56.941172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:56.941079    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:57.044494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:57.044464    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"]
Nov 01 09:03:57.045323 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:03:57.045296    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47544b66_9c4e_4999_b3ff_0441163e0b93.slice/crio-cb70a8eb3c734a76e0f92a24d7784c5de1382b07ae35fde1f13c8ec0825c1ee5 WatchSource:0}: Error finding container cb70a8eb3c734a76e0f92a24d7784c5de1382b07ae35fde1f13c8ec0825c1ee5: Status 404 returned error can't find the container with id cb70a8eb3c734a76e0f92a24d7784c5de1382b07ae35fde1f13c8ec0825c1ee5
Nov 01 09:03:57.047348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:57.047328    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:03:57.133826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:57.133797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:57.342940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:57.342904    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:57.544886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:57.544852    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:03:57.561450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:57.561418    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb" event={"ID":"47544b66-9c4e-4999-b3ff-0441163e0b93","Type":"ContainerStarted","Data":"cb70a8eb3c734a76e0f92a24d7784c5de1382b07ae35fde1f13c8ec0825c1ee5"}
Nov 01 09:03:58.226970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:03:58.226936    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:00.350425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.350389    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:00.606497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.606414    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:00.753734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.753700    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"]
Nov 01 09:04:00.757638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.757612    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:00.853321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.853288    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-095909-dockercfg-5jz64\""
Nov 01 09:04:00.853986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.853961    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-095909\""
Nov 01 09:04:00.910295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.886660    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"]
Nov 01 09:04:00.910295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.888220    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:00.910295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.888343    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbrz\" (UniqueName: \"kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz\") pod \"payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:00.989519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.989475    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:00.989731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.989608    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbrz\" (UniqueName: \"kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz\") pod \"payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:00.990621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:00.990519    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:01.047128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.047088    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbrz\" (UniqueName: \"kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz\") pod \"payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:01.071929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.071890    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:01.447832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.447795    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"]
Nov 01 09:04:01.450239 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:04:01.450210    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf666f42_b43a_480e_a095_2f162a34be3e.slice/crio-fe6a735465f1142c1f14de9681c0259e9a9a76114fa099067a2674aa43d46fe8 WatchSource:0}: Error finding container fe6a735465f1142c1f14de9681c0259e9a9a76114fa099067a2674aa43d46fe8: Status 404 returned error can't find the container with id fe6a735465f1142c1f14de9681c0259e9a9a76114fa099067a2674aa43d46fe8
Nov 01 09:04:01.592080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.592036    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf" event={"ID":"af666f42-b43a-480e-a095-2f162a34be3e","Type":"ContainerStarted","Data":"29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2"}
Nov 01 09:04:01.592322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.592106    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf" event={"ID":"af666f42-b43a-480e-a095-2f162a34be3e","Type":"ContainerStarted","Data":"fe6a735465f1142c1f14de9681c0259e9a9a76114fa099067a2674aa43d46fe8"}
Nov 01 09:04:01.592322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.592130    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:01.701834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:01.701756    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf" podStartSLOduration=1.701742527 podStartE2EDuration="1.701742527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:01.699443646 +0000 UTC m=+1914.893838202" watchObservedRunningTime="2026-11-01 09:04:01.701742527 +0000 UTC m=+1914.896137133"
Nov 01 09:04:02.632577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:02.628088    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb" event={"ID":"47544b66-9c4e-4999-b3ff-0441163e0b93","Type":"ContainerStarted","Data":"171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1"}
Nov 01 09:04:02.632577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:02.630135    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:04:02.739073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:02.739015    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb" podStartSLOduration=2.138056059 podStartE2EDuration="6.739000101s" totalImagesPullingTime="4.600944042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:02.738568473 +0000 UTC m=+1915.932963028" watchObservedRunningTime="2026-11-01 09:04:02.739000101 +0000 UTC m=+1915.933394655"
Nov 01 09:04:02.877073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:02.877038    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:03.512443 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:03.512403    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:08.025158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:08.025125    4314 scope.go:122] "RemoveContainer" containerID="7e01712a2eaed4179612fc69bf3597057ceef98dc3578040a2aa27f84a844d3e"
Nov 01 09:04:08.036870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:08.036844    4314 scope.go:122] "RemoveContainer" containerID="9bfa5e5f12ac816dc104e4968114fdab24fd53553f431f1b76576e09bc8dea52"
Nov 01 09:04:08.702058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:08.702010    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:10.847545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:10.847481    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:11.323739 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:04:11.323700    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:04:12.286223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:12.286189    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:12.626520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:12.626442    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:04:12.784951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:12.784913    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:14.638274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:14.638243    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:04:14.788734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:14.788695    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:16.591831 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:16.591794    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:18.791928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:18.791844    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:19.649881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:19.649842    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:19.858166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:19.858118    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:21.496400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:21.496362    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:22.807285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:22.807235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:22.973965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:22.973930    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:23.324035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:23.323985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:23.490674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:23.490635    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:23.670248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:23.670164    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:24.768360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:24.768325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:25.223838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:25.223719    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:26.189240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:26.189194    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:26.652728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:26.652685    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:26.818836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:26.818801    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:26.994413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:26.994377    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:27.144116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.144079    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:27.279024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.278941    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"]
Nov 01 09:04:27.284795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.284773    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.297672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.297649    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:27.337352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.337323    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"]
Nov 01 09:04:27.342612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.342523    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.342761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.342679    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nkp\" (UniqueName: \"kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.383622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.378342    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb28ec-dockercfg-z9rk7\""
Nov 01 09:04:27.383622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.378890    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb28ec\""
Nov 01 09:04:27.458644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.449661    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.458644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.449769    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nkp\" (UniqueName: \"kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.458644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.451007    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.507427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.507398    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nkp\" (UniqueName: \"kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.598825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.598741    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:27.970866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:27.970822    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"]
Nov 01 09:04:27.972059 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:04:27.972029    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8071338b_a7a1_4741_810f_4c7d90400e3d.slice/crio-3a79498802dc0d95b655b19025a75fe90d7eef9fbe5679a1b384b1d132f605ae WatchSource:0}: Error finding container 3a79498802dc0d95b655b19025a75fe90d7eef9fbe5679a1b384b1d132f605ae: Status 404 returned error can't find the container with id 3a79498802dc0d95b655b19025a75fe90d7eef9fbe5679a1b384b1d132f605ae
Nov 01 09:04:28.816205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:28.816166    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx" event={"ID":"8071338b-a7a1-4741-810f-4c7d90400e3d","Type":"ContainerStarted","Data":"9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28"}
Nov 01 09:04:28.816663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:28.816233    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx" event={"ID":"8071338b-a7a1-4741-810f-4c7d90400e3d","Type":"ContainerStarted","Data":"3a79498802dc0d95b655b19025a75fe90d7eef9fbe5679a1b384b1d132f605ae"}
Nov 01 09:04:28.816663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:28.816256    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:28.927246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:28.927197    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx" podStartSLOduration=1.927180693 podStartE2EDuration="1.927180693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:28.925258624 +0000 UTC m=+1942.119653177" watchObservedRunningTime="2026-11-01 09:04:28.927180693 +0000 UTC m=+1942.121575245"
Nov 01 09:04:31.617178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:31.617145    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:31.839268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:31.839235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:31.990385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:31.990351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:32.336495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:32.336375    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:35.914765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:35.914720    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:36.069982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:36.069907    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:37.588991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:37.588955    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:39.828923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:39.828887    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:04:41.323345 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:04:41.323311    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:04:43.130889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:43.130849    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:49.813115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:49.813078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:49.987820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:49.987771    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:50.135354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:50.135277    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:52.427699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:52.427656    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:04:52.428144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:52.427727    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:04:54.970977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:54.970898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:55.025853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:55.025820    4314 generic.go:356] "Generic (PLEG): container finished" podID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerID="a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c" exitCode=0
Nov 01 09:04:55.026061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:55.025898    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" event={"ID":"9d60b6bf-33f2-4162-a2d7-13ace4e85faa","Type":"ContainerDied","Data":"a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"}
Nov 01 09:04:55.026416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:55.026391    4314 scope.go:122] "RemoveContainer" containerID="a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"
Nov 01 09:04:55.028648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:55.028617    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:04:55.028733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:55.028686    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:04:55.028778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:55.028738    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:04:56.034542 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:56.034493    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" event={"ID":"9d60b6bf-33f2-4162-a2d7-13ace4e85faa","Type":"ContainerStarted","Data":"307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a"}
Nov 01 09:04:56.932298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:56.932261    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:04:56.932482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:56.932463    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 01 09:04:56.932555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:56.932510    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 01 09:04:57.520466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:57.520425    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:57.677992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:57.677938    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:57.932024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:57.931951    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 01 09:04:57.932024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:57.931994    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 01 09:04:58.931782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:58.931743    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 01 09:04:58.932267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:58.931798    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 01 09:04:59.780021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:59.779984    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:04:59.931612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:59.931576    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 01 09:04:59.931787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:04:59.931635    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 01 09:05:00.932290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:00.932244    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 01 09:05:00.932810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:00.932308    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 01 09:05:01.935271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:01.935239    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:05:01.935707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:01.935683    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:05:01.936211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:01.936195    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:05:04.724448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:04.724406    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:08.084190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:08.084156    4314 scope.go:122] "RemoveContainer" containerID="f764643204b3abd41554a01a0f1719fab61b10a7a2f24289c12562a6bab775fe"
Nov 01 09:05:08.340648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:08.340494    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:08.491629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:08.491511    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:11.241202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:11.241167    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:11.323713 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:11.323679    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:05:11.394309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:11.394273    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:13.625187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:13.625151    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"]
Nov 01 09:05:13.625700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:13.625387    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy" containerID="cri-o://36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61" gracePeriod=30
Nov 01 09:05:13.672199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:13.672159    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"]
Nov 01 09:05:13.672453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:13.672426    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy" containerID="cri-o://e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5" gracePeriod=30
Nov 01 09:05:15.996834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:15.996803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:18.929315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:18.929144    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 09:05:18.971258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:18.971231    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 09:05:19.062689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062656    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.062886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062698    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.062886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062728    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.062886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062763    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.062886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062807    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.062886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062832    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.062886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062864    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062913    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062952    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062989    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.062999    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063059    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063053    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063087    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063062    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063128    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063141    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063171    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063221    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.063933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063350    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z\") pod \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\" (UID: \"a35b7a28-8f7d-477f-8584-e999ac50b0dd\") "
Nov 01 09:05:19.063933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063409    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.063933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063448    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.063933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063462    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84\") pod \"b029da95-7787-4a8e-93af-e74a100f7fb3\" (UID: \"b029da95-7787-4a8e-93af-e74a100f7fb3\") "
Nov 01 09:05:19.063933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.063802    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064247    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064273    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064298    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064313    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064328    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064343    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064470    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.064718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064591    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.065225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064734    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:19.065225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.064793    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.065974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.065948    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:19.066083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.065952    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.066083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.065986    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:19.066083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.066065    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:19.066211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.066110    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:19.066355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.066336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84" pod "b029da95-7787-4a8e-93af-e74a100f7fb3" (UID: "b029da95-7787-4a8e-93af-e74a100f7fb3"). InnerVolumeSpecName "kube-api-access-gcr84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:19.067030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.067003    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:19.067232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.067214    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z" pod "a35b7a28-8f7d-477f-8584-e999ac50b0dd" (UID: "a35b7a28-8f7d-477f-8584-e999ac50b0dd"). InnerVolumeSpecName "kube-api-access-qdq2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165904    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165934    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165944    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165953    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165963    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165971    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b029da95-7787-4a8e-93af-e74a100f7fb3-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165981    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165989    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a35b7a28-8f7d-477f-8584-e999ac50b0dd-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.165990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.165997    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b029da95-7787-4a8e-93af-e74a100f7fb3-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.166394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.166008    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdq2z\" (UniqueName: \"kubernetes.io/projected/a35b7a28-8f7d-477f-8584-e999ac50b0dd-kube-api-access-qdq2z\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.166394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.166018    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b029da95-7787-4a8e-93af-e74a100f7fb3-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.166394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.166026    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcr84\" (UniqueName: \"kubernetes.io/projected/b029da95-7787-4a8e-93af-e74a100f7fb3-kube-api-access-gcr84\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:19.217684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.217652    4314 generic.go:356] "Generic (PLEG): container finished" podID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerID="e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5" exitCode=0
Nov 01 09:05:19.217879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.217717    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"
Nov 01 09:05:19.217879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.217727    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" event={"ID":"b029da95-7787-4a8e-93af-e74a100f7fb3","Type":"ContainerDied","Data":"e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5"}
Nov 01 09:05:19.217879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.217785    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj" event={"ID":"b029da95-7787-4a8e-93af-e74a100f7fb3","Type":"ContainerDied","Data":"05dfd20d1243b2429d44510d333f5742338a1799685c95be288dac0b5d1e7375"}
Nov 01 09:05:19.217879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.217815    4314 scope.go:122] "RemoveContainer" containerID="e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5"
Nov 01 09:05:19.219738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.219702    4314 generic.go:356] "Generic (PLEG): container finished" podID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerID="36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61" exitCode=0
Nov 01 09:05:19.219909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.219765    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"
Nov 01 09:05:19.219909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.219775    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" event={"ID":"a35b7a28-8f7d-477f-8584-e999ac50b0dd","Type":"ContainerDied","Data":"36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61"}
Nov 01 09:05:19.219909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.219817    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm" event={"ID":"a35b7a28-8f7d-477f-8584-e999ac50b0dd","Type":"ContainerDied","Data":"103fcf043ae4e4ab0ea43e623c444b71c7a21f938b8ccd5ef604492a1cdb4f82"}
Nov 01 09:05:19.229356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.229330    4314 scope.go:122] "RemoveContainer" containerID="e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5"
Nov 01 09:05:19.229871 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:19.229841    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5\": container with ID starting with e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5 not found: ID does not exist" containerID="e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5"
Nov 01 09:05:19.229970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.229881    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5"} err="failed to get container status \"e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5\": rpc error: code = NotFound desc = could not find container \"e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5\": container with ID starting with e2cd3dd6105d1112436840e2cc84613180fefc8ea7141f94f77880562f1917b5 not found: ID does not exist"
Nov 01 09:05:19.229970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.229905    4314 scope.go:122] "RemoveContainer" containerID="36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61"
Nov 01 09:05:19.241307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.241282    4314 scope.go:122] "RemoveContainer" containerID="36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61"
Nov 01 09:05:19.241632 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:19.241609    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61\": container with ID starting with 36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61 not found: ID does not exist" containerID="36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61"
Nov 01 09:05:19.241711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.241639    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61"} err="failed to get container status \"36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61\": rpc error: code = NotFound desc = could not find container \"36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61\": container with ID starting with 36eb84e7236511c5174e36e6d008956c289bd265d1f28d04fc3210fa27f5ae61 not found: ID does not exist"
Nov 01 09:05:19.373744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.373706    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"]
Nov 01 09:05:19.377905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.377877    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-5vbjj"]
Nov 01 09:05:19.480107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.480080    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"]
Nov 01 09:05:19.489243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:19.489213    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-2tsvm"]
Nov 01 09:05:20.733824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:20.733789    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:20.893843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:20.893808    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:20.929747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:20.929708    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" path="/var/lib/kubelet/pods/a35b7a28-8f7d-477f-8584-e999ac50b0dd/volumes"
Nov 01 09:05:20.930287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:20.930264    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" path="/var/lib/kubelet/pods/b029da95-7787-4a8e-93af-e74a100f7fb3/volumes"
Nov 01 09:05:21.141198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:21.141112    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:21.503423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:21.503388    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:21.650215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:21.650181    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:25.775709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:25.775675    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:28.206412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.206375    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:28.740600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.740560    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"]
Nov 01 09:05:28.741095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.741082    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy"
Nov 01 09:05:28.741139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.741097    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy"
Nov 01 09:05:28.741139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.741121    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy"
Nov 01 09:05:28.741139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.741127    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy"
Nov 01 09:05:28.741281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.741222    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b029da95-7787-4a8e-93af-e74a100f7fb3" containerName="istio-proxy"
Nov 01 09:05:28.741281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.741233    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35b7a28-8f7d-477f-8584-e999ac50b0dd" containerName="istio-proxy"
Nov 01 09:05:28.745878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.745860    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.795590    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-eaabf4-openshift-default-dockercfg-8xfcd\""
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802276    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802326    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802479    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802547    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xwp\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802598    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802634    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802701    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802738    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.803649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.802778    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.830776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.829837    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:28.860472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.860435    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"]
Nov 01 09:05:28.904636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904595    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.904842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904677    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xwp\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.904842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904720    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.904842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904746    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.904842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904807    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.904842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904843    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.905753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904883    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.905753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904917    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.905753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.904943    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.905753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.905033    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.905753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.905282    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.905753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.905587    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.906000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.905878    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.906000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.905962    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.907882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.907860    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.910213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.910193    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.982410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.982382    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:28.982900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:28.982878    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xwp\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp\") pod \"e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") " pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:29.055995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:29.055907    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:29.067018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:29.066976    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:29.489642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:29.489615    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"]
Nov 01 09:05:29.490169 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:05:29.490144    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258e5053_6b33_47e6_8cee_ba4f4f1a0983.slice/crio-2dbaf6dcf8114931735d9e5654714b706724f639ee63332f44903c35194533a6 WatchSource:0}: Error finding container 2dbaf6dcf8114931735d9e5654714b706724f639ee63332f44903c35194533a6: Status 404 returned error can't find the container with id 2dbaf6dcf8114931735d9e5654714b706724f639ee63332f44903c35194533a6
Nov 01 09:05:29.492801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:29.492764    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:05:29.492907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:29.492845    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:05:29.492907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:29.492893    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:05:30.290607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:30.290555    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" event={"ID":"258e5053-6b33-47e6-8cee-ba4f4f1a0983","Type":"ContainerStarted","Data":"cf1fec34c259b82aace395edff508246620565fb56f26e84cbdface393ddc5f5"}
Nov 01 09:05:30.290805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:30.290634    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" event={"ID":"258e5053-6b33-47e6-8cee-ba4f4f1a0983","Type":"ContainerStarted","Data":"2dbaf6dcf8114931735d9e5654714b706724f639ee63332f44903c35194533a6"}
Nov 01 09:05:30.500666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:30.500602    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podStartSLOduration=2.500577906 podStartE2EDuration="2.500577906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:05:30.452340678 +0000 UTC m=+2003.646735230" watchObservedRunningTime="2026-11-01 09:05:30.500577906 +0000 UTC m=+2003.694972460"
Nov 01 09:05:30.574393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:30.574310    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:30.735654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:30.735624    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:30.890493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:30.890407    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:31.067551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.067512    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:31.073568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.073522    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:31.297514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.297480    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:31.298504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.298484    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:05:31.819711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.819669    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"]
Nov 01 09:05:31.820205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.819877    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" containerID="cri-o://2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf" gracePeriod=30
Nov 01 09:05:31.823153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.823122    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"]
Nov 01 09:05:31.823372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:31.823346    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" containerID="cri-o://6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674" gracePeriod=30
Nov 01 09:05:33.394737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:33.394703    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:33.565577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:33.565519    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:33.714696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:33.714594    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:35.519991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:35.519953    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:05:35.520397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:35.520013    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:05:36.053992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:36.053949    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:36.294047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:36.294007    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:36.430455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:36.430370    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:05:36.430455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:36.430431    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:05:37.097211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.097140    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 09:05:37.120489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.120466    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 09:05:37.208555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208497    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.208555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208559    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.208800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208613    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.208800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208633    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.208800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208705    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.208800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208748    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208835    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.209027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208898    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.209027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208952    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.209027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.208989    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209030    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209073    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209100    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.209249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209146    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs\") pod \"433f22be-eed2-465f-a782-ecdb78dee65d\" (UID: \"433f22be-eed2-465f-a782-ecdb78dee65d\") "
Nov 01 09:05:37.209249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209195    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209264    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209304    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209322    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket\") pod \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\" (UID: \"6e6963b4-79b3-488f-a6c3-3fceacb5b459\") "
Nov 01 09:05:37.209515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209322    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.209778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209676    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:37.209843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209781    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f22be-eed2-465f-a782-ecdb78dee65d-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.209843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209795    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.209964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209938    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.210022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.209960    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.210142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.210121    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.210418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.210396    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:37.210689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.210652    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.210881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.210862    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.211109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211068    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.211287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211234    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.211372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211335    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "kube-api-access-lrblz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.211372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211340    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:37.211700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211663    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.211818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211755    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "kube-api-access-czfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.211916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.211897    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:37.212973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.212950    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy" pod "433f22be-eed2-465f-a782-ecdb78dee65d" (UID: "433f22be-eed2-465f-a782-ecdb78dee65d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.212973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.212957    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.213228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.213213    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token" pod "6e6963b4-79b3-488f-a6c3-3fceacb5b459" (UID: "6e6963b4-79b3-488f-a6c3-3fceacb5b459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.311082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311050    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311077    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311087    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311096    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311105    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f22be-eed2-465f-a782-ecdb78dee65d-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311113    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311123    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrblz\" (UniqueName: \"kubernetes.io/projected/6e6963b4-79b3-488f-a6c3-3fceacb5b459-kube-api-access-lrblz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311133    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/433f22be-eed2-465f-a782-ecdb78dee65d-kube-api-access-czfnb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311144    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311152    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311160    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311168    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311176    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e6963b4-79b3-488f-a6c3-3fceacb5b459-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311184    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311192    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f22be-eed2-465f-a782-ecdb78dee65d-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.311341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.311200    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e6963b4-79b3-488f-a6c3-3fceacb5b459-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.335557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.335500    4314 generic.go:356] "Generic (PLEG): container finished" podID="433f22be-eed2-465f-a782-ecdb78dee65d" containerID="6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674" exitCode=0
Nov 01 09:05:37.335750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.335574    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"
Nov 01 09:05:37.335750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.335588    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" event={"ID":"433f22be-eed2-465f-a782-ecdb78dee65d","Type":"ContainerDied","Data":"6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674"}
Nov 01 09:05:37.335750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.335654    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7" event={"ID":"433f22be-eed2-465f-a782-ecdb78dee65d","Type":"ContainerDied","Data":"f9d3a6650b85625a2f895beec66bfc784ac2285e9c0ecc8116d52c3a9cf3c837"}
Nov 01 09:05:37.335750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.335685    4314 scope.go:122] "RemoveContainer" containerID="6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674"
Nov 01 09:05:37.337480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.337455    4314 generic.go:356] "Generic (PLEG): container finished" podID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerID="2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf" exitCode=0
Nov 01 09:05:37.337623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.337508    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"
Nov 01 09:05:37.337715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.337512    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" event={"ID":"6e6963b4-79b3-488f-a6c3-3fceacb5b459","Type":"ContainerDied","Data":"2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf"}
Nov 01 09:05:37.337802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.337781    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549" event={"ID":"6e6963b4-79b3-488f-a6c3-3fceacb5b459","Type":"ContainerDied","Data":"6c3e5f327fd9b9de5781cd16452645a2813568d72c0d48ccc89b443a56350553"}
Nov 01 09:05:37.349737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.349716    4314 scope.go:122] "RemoveContainer" containerID="6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674"
Nov 01 09:05:37.350293 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:37.350227    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674\": container with ID starting with 6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674 not found: ID does not exist" containerID="6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674"
Nov 01 09:05:37.350293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.350269    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674"} err="failed to get container status \"6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674\": rpc error: code = NotFound desc = could not find container \"6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674\": container with ID starting with 6046280339a078a99940918c5ce8bed2fdaf91cc4146894660243e99c4e29674 not found: ID does not exist"
Nov 01 09:05:37.350293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.350294    4314 scope.go:122] "RemoveContainer" containerID="2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf"
Nov 01 09:05:37.362082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.362056    4314 scope.go:122] "RemoveContainer" containerID="2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf"
Nov 01 09:05:37.362444 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:37.362415    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf\": container with ID starting with 2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf not found: ID does not exist" containerID="2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf"
Nov 01 09:05:37.362626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.362460    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf"} err="failed to get container status \"2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf\": rpc error: code = NotFound desc = could not find container \"2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf\": container with ID starting with 2081da0f5497985755e283ab4d4d6165ac876cf383a8b4b07bf6f0694e666faf not found: ID does not exist"
Nov 01 09:05:37.481379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.481344    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"]
Nov 01 09:05:37.485850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.485819    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-m6549"]
Nov 01 09:05:37.587686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.587653    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"]
Nov 01 09:05:37.590409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.590388    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6hwx7"]
Nov 01 09:05:37.865871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:37.865837    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:38.220875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.220788    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:38.391631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.391577    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:38.635704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.635671    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"]
Nov 01 09:05:38.636268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.636247    4314 cpu_manager.go:414] "containerMap: removing container" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy"
Nov 01 09:05:38.636393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.636268    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy"
Nov 01 09:05:38.636393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.636311    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy"
Nov 01 09:05:38.636393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.636321    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy"
Nov 01 09:05:38.636568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.636477    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" containerName="istio-proxy"
Nov 01 09:05:38.636568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.636494    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" containerName="istio-proxy"
Nov 01 09:05:38.641229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.641206    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.691072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.691037    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eaabf4-dockercfg-sv875\""
Nov 01 09:05:38.761555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.759862    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eaabf4\""
Nov 01 09:05:38.761754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.761560    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtlp\" (UniqueName: \"kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.761754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.761700    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.772562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.771694    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"]
Nov 01 09:05:38.786775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.782450    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"]
Nov 01 09:05:38.805493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.803852    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:38.834620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.834583    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"]
Nov 01 09:05:38.862475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.862442    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:38.862679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.862640    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtlp\" (UniqueName: \"kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.862746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.862717    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7fn\" (UniqueName: \"kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn\") pod \"payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:38.862907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.862883    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.863589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.863571    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.921735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.921655    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtlp\" (UniqueName: \"kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.928807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.928778    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433f22be-eed2-465f-a782-ecdb78dee65d" path="/var/lib/kubelet/pods/433f22be-eed2-465f-a782-ecdb78dee65d/volumes"
Nov 01 09:05:38.929250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.929236    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6963b4-79b3-488f-a6c3-3fceacb5b459" path="/var/lib/kubelet/pods/6e6963b4-79b3-488f-a6c3-3fceacb5b459/volumes"
Nov 01 09:05:38.954917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.954891    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:38.964370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.964346    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7fn\" (UniqueName: \"kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn\") pod \"payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:38.964483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.964467    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:38.965104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:38.965087    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:39.028348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:39.028258    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7fn\" (UniqueName: \"kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn\") pod \"payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:39.128139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:39.128106    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:39.341385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:39.341352    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"]
Nov 01 09:05:39.341975 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:05:39.341940    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb172518_dee4_4b68_a05d_1a4ad4671d18.slice/crio-840fba7fe45e615d5d346c6bff17deb4d7a4feae10fe32b5f40b3c2e92b22a3b WatchSource:0}: Error finding container 840fba7fe45e615d5d346c6bff17deb4d7a4feae10fe32b5f40b3c2e92b22a3b: Status 404 returned error can't find the container with id 840fba7fe45e615d5d346c6bff17deb4d7a4feae10fe32b5f40b3c2e92b22a3b
Nov 01 09:05:39.390568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:39.390512    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m" event={"ID":"fb172518-dee4-4b68-a05d-1a4ad4671d18","Type":"ContainerStarted","Data":"840fba7fe45e615d5d346c6bff17deb4d7a4feae10fe32b5f40b3c2e92b22a3b"}
Nov 01 09:05:39.496870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:39.496838    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"]
Nov 01 09:05:39.497983 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:05:39.497959    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fccc27_83f0_43cf_b3a5_511a4723a371.slice/crio-fcbd3133205272cc34a1b3501e8f81ba5929b50524564e5016023eb10c2911eb WatchSource:0}: Error finding container fcbd3133205272cc34a1b3501e8f81ba5929b50524564e5016023eb10c2911eb: Status 404 returned error can't find the container with id fcbd3133205272cc34a1b3501e8f81ba5929b50524564e5016023eb10c2911eb
Nov 01 09:05:40.398296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.398261    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz" event={"ID":"f7fccc27-83f0-43cf-b3a5-511a4723a371","Type":"ContainerStarted","Data":"85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023"}
Nov 01 09:05:40.398772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.398320    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz" event={"ID":"f7fccc27-83f0-43cf-b3a5-511a4723a371","Type":"ContainerStarted","Data":"fcbd3133205272cc34a1b3501e8f81ba5929b50524564e5016023eb10c2911eb"}
Nov 01 09:05:40.398772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.398362    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:40.400212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.400181    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m" event={"ID":"fb172518-dee4-4b68-a05d-1a4ad4671d18","Type":"ContainerStarted","Data":"eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995"}
Nov 01 09:05:40.400346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.400237    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:40.515000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.514947    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz" podStartSLOduration=2.514931154 podStartE2EDuration="2.514931154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:05:40.512325921 +0000 UTC m=+2013.706720476" watchObservedRunningTime="2026-11-01 09:05:40.514931154 +0000 UTC m=+2013.709325709"
Nov 01 09:05:40.641114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:40.641046    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m" podStartSLOduration=2.641026038 podStartE2EDuration="2.641026038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:05:40.627976302 +0000 UTC m=+2013.822370859" watchObservedRunningTime="2026-11-01 09:05:40.641026038 +0000 UTC m=+2013.835420595"
Nov 01 09:05:41.322322 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:41.322286    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:05:41.329177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:41.329149    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:41.573297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:41.573213    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"]
Nov 01 09:05:41.671182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:41.671145    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"]
Nov 01 09:05:41.921397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:41.921319    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:42.084906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:42.084871    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:44.876945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:44.876899    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:46.046691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.046658    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"]
Nov 01 09:05:46.093706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.093670    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-287ebe29-l8dpk"]
Nov 01 09:05:46.574459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.574420    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67" podUID="f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" containerName="payload-processing" containerID="cri-o://442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b" gracePeriod=25
Nov 01 09:05:46.671688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.671632    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt" podUID="4a898fed-a7df-4614-88b5-62056f21ed9a" containerName="payload-pre-processing" containerID="cri-o://db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2" gracePeriod=25
Nov 01 09:05:46.760314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.760267    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:46.862680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.862658    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:05:46.930725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.930696    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97bf487d-ec2a-4038-9337-51331994ef7c" path="/var/lib/kubelet/pods/97bf487d-ec2a-4038-9337-51331994ef7c/volumes"
Nov 01 09:05:46.950776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.950750    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:05:46.986117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.986081    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume\") pod \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") "
Nov 01 09:05:46.986318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.986298    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk\" (UniqueName: \"kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk\") pod \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\" (UID: \"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4\") "
Nov 01 09:05:46.986498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.986474    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume" pod "f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" (UID: "f5b08a1b-634d-4d56-bfa9-83bcce0de3e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:46.987004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.986977    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:46.988442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:46.988418    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk" pod "f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" (UID: "f5b08a1b-634d-4d56-bfa9-83bcce0de3e4"). InnerVolumeSpecName "kube-api-access-9wcwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:47.087948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.087860    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume\") pod \"4a898fed-a7df-4614-88b5-62056f21ed9a\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") "
Nov 01 09:05:47.087948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.087925    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf\" (UniqueName: \"kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf\") pod \"4a898fed-a7df-4614-88b5-62056f21ed9a\" (UID: \"4a898fed-a7df-4614-88b5-62056f21ed9a\") "
Nov 01 09:05:47.088477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.088205    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume" pod "4a898fed-a7df-4614-88b5-62056f21ed9a" (UID: "4a898fed-a7df-4614-88b5-62056f21ed9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:47.088793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.088770    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wcwk\" (UniqueName: \"kubernetes.io/projected/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4-kube-api-access-9wcwk\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:47.088941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.088926    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a898fed-a7df-4614-88b5-62056f21ed9a-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:47.090295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.090276    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf" pod "4a898fed-a7df-4614-88b5-62056f21ed9a" (UID: "4a898fed-a7df-4614-88b5-62056f21ed9a"). InnerVolumeSpecName "kube-api-access-4bdcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:47.190051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.190010    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bdcf\" (UniqueName: \"kubernetes.io/projected/4a898fed-a7df-4614-88b5-62056f21ed9a-kube-api-access-4bdcf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:47.447982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.447888    4314 generic.go:356] "Generic (PLEG): container finished" podID="f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" containerID="442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b" exitCode=0
Nov 01 09:05:47.447982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.447944    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"
Nov 01 09:05:47.448226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.447977    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67" event={"ID":"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4","Type":"ContainerDied","Data":"442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b"}
Nov 01 09:05:47.448226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.448038    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67" event={"ID":"f5b08a1b-634d-4d56-bfa9-83bcce0de3e4","Type":"ContainerDied","Data":"a86a74413abac407dd034c4d8ce1f3ca9e6db89dfee670cf3d1d6af1c22c8129"}
Nov 01 09:05:47.448226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.448069    4314 scope.go:122] "RemoveContainer" containerID="442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b"
Nov 01 09:05:47.449593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.449555    4314 generic.go:356] "Generic (PLEG): container finished" podID="4a898fed-a7df-4614-88b5-62056f21ed9a" containerID="db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2" exitCode=0
Nov 01 09:05:47.449593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.449587    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt" event={"ID":"4a898fed-a7df-4614-88b5-62056f21ed9a","Type":"ContainerDied","Data":"db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2"}
Nov 01 09:05:47.449782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.449597    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"
Nov 01 09:05:47.449782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.449629    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt" event={"ID":"4a898fed-a7df-4614-88b5-62056f21ed9a","Type":"ContainerDied","Data":"983a15293afbc3f391140d23d11ca9c3932ba1c724da11080448d210395c3761"}
Nov 01 09:05:47.458829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.458789    4314 scope.go:122] "RemoveContainer" containerID="442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b"
Nov 01 09:05:47.459186 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:47.459163    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b\": container with ID starting with 442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b not found: ID does not exist" containerID="442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b"
Nov 01 09:05:47.459285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.459197    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b"} err="failed to get container status \"442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b\": rpc error: code = NotFound desc = could not find container \"442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b\": container with ID starting with 442ec5842c14e7ba92b025c02b4550cd1d31a71ccaad1fe5a5b5447309e7f97b not found: ID does not exist"
Nov 01 09:05:47.459285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.459224    4314 scope.go:122] "RemoveContainer" containerID="db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2"
Nov 01 09:05:47.468649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.468628    4314 scope.go:122] "RemoveContainer" containerID="db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2"
Nov 01 09:05:47.469928 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:05:47.469904    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2\": container with ID starting with db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2 not found: ID does not exist" containerID="db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2"
Nov 01 09:05:47.470032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.469940    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2"} err="failed to get container status \"db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2\": rpc error: code = NotFound desc = could not find container \"db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2\": container with ID starting with db4621fa83c3098bbba29d69549d3e7ea96530465af097216bc2bf93ee571ee2 not found: ID does not exist"
Nov 01 09:05:47.602748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.602719    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"]
Nov 01 09:05:47.648854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.648822    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-65cf6cfbf5-7lmrt"]
Nov 01 09:05:47.709775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.709685    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"]
Nov 01 09:05:47.713412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:47.713376    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-768bfd484c-9hp67"]
Nov 01 09:05:48.928505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:48.928429    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a898fed-a7df-4614-88b5-62056f21ed9a" path="/var/lib/kubelet/pods/4a898fed-a7df-4614-88b5-62056f21ed9a/volumes"
Nov 01 09:05:48.928886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:48.928861    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" path="/var/lib/kubelet/pods/f5b08a1b-634d-4d56-bfa9-83bcce0de3e4/volumes"
Nov 01 09:05:51.411720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:51.411688    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:05:51.412158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:51.411764    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:05:53.722958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:53.722924    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:05:58.977449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:05:58.977415    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:00.120357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:00.120321    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:00.978904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:00.978870    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:01.506973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.506935    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"]
Nov 01 09:06:01.507448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.507434    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" containerName="payload-processing"
Nov 01 09:06:01.507493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.507449    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" containerName="payload-processing"
Nov 01 09:06:01.507493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.507465    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a898fed-a7df-4614-88b5-62056f21ed9a" containerName="payload-pre-processing"
Nov 01 09:06:01.507493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.507471    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a898fed-a7df-4614-88b5-62056f21ed9a" containerName="payload-pre-processing"
Nov 01 09:06:01.507636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.507622    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b08a1b-634d-4d56-bfa9-83bcce0de3e4" containerName="payload-processing"
Nov 01 09:06:01.507675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.507639    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a898fed-a7df-4614-88b5-62056f21ed9a" containerName="payload-pre-processing"
Nov 01 09:06:01.510998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.510982    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570245    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570329    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570391    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570426    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570466    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570490    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570515    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570625    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.570710    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7cc\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.572580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.571076    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c0460d-openshift-default-dockercfg-9mk9p\""
Nov 01 09:06:01.609579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.599887    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:01.622054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.622011    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"]
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673149    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673273    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673345    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673387    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673432    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673462    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673499    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673663    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.674100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.673779    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7cc\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.675244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.674971    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.680176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.676986    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.680176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.677474    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.681036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.681011    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.687893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.687856    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.706246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.706207    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.706907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.706880    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.731722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.731685    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.731897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.731851    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7cc\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc\") pod \"e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") " pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.843915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.843810    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:01.846697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:01.846668    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:02.197742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.197655    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"]
Nov 01 09:06:02.318262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.318235    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"]
Nov 01 09:06:02.319115 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:06:02.319088    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fef78d8_13c5_4b75_8a1f_f769fe874167.slice/crio-752eb59e42d8f2591f401466e77ae08ed6695b415d21eb5d963256884c273f6c WatchSource:0}: Error finding container 752eb59e42d8f2591f401466e77ae08ed6695b415d21eb5d963256884c273f6c: Status 404 returned error can't find the container with id 752eb59e42d8f2591f401466e77ae08ed6695b415d21eb5d963256884c273f6c
Nov 01 09:06:02.321597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.321559    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:06:02.321718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.321629    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:06:02.321718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.321660    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:06:02.437149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.437113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:02.558386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.558338    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" event={"ID":"0fef78d8-13c5-4b75-8a1f-f769fe874167","Type":"ContainerStarted","Data":"c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"}
Nov 01 09:06:02.558998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.558412    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" event={"ID":"0fef78d8-13c5-4b75-8a1f-f769fe874167","Type":"ContainerStarted","Data":"752eb59e42d8f2591f401466e77ae08ed6695b415d21eb5d963256884c273f6c"}
Nov 01 09:06:02.579737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.579691    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:02.674656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.674595    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podStartSLOduration=1.674577183 podStartE2EDuration="1.674577183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:02.672629449 +0000 UTC m=+2035.867024008" watchObservedRunningTime="2026-11-01 09:06:02.674577183 +0000 UTC m=+2035.868971739"
Nov 01 09:06:02.844046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:02.843966    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:03.850462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:03.850433    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:04.378159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:04.378125    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:04.571367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:04.571341    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:04.572523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:04.572498    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:06:06.375649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:06.375613    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:06.548741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:06.548706    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:06.737892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:06.737857    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:07.198839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.198756    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf" podUID="af666f42-b43a-480e-a095-2f162a34be3e" containerName="payload-pre-processing" containerID="cri-o://29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2" gracePeriod=25
Nov 01 09:06:07.453295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.453235    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:06:07.563395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.563358    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz\" (UniqueName: \"kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz\") pod \"af666f42-b43a-480e-a095-2f162a34be3e\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") "
Nov 01 09:06:07.563622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.563411    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume\") pod \"af666f42-b43a-480e-a095-2f162a34be3e\" (UID: \"af666f42-b43a-480e-a095-2f162a34be3e\") "
Nov 01 09:06:07.564060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.563965    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume" pod "af666f42-b43a-480e-a095-2f162a34be3e" (UID: "af666f42-b43a-480e-a095-2f162a34be3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:06:07.566004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.565976    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz" pod "af666f42-b43a-480e-a095-2f162a34be3e" (UID: "af666f42-b43a-480e-a095-2f162a34be3e"). InnerVolumeSpecName "kube-api-access-cbbrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:06:07.591441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.591409    4314 generic.go:356] "Generic (PLEG): container finished" podID="af666f42-b43a-480e-a095-2f162a34be3e" containerID="29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2" exitCode=0
Nov 01 09:06:07.591589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.591453    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"
Nov 01 09:06:07.591589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.591493    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf" event={"ID":"af666f42-b43a-480e-a095-2f162a34be3e","Type":"ContainerDied","Data":"29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2"}
Nov 01 09:06:07.591722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.591590    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf" event={"ID":"af666f42-b43a-480e-a095-2f162a34be3e","Type":"ContainerDied","Data":"fe6a735465f1142c1f14de9681c0259e9a9a76114fa099067a2674aa43d46fe8"}
Nov 01 09:06:07.591722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.591641    4314 scope.go:122] "RemoveContainer" containerID="29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2"
Nov 01 09:06:07.601918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.601897    4314 scope.go:122] "RemoveContainer" containerID="29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2"
Nov 01 09:06:07.605701 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:06:07.602817    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2\": container with ID starting with 29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2 not found: ID does not exist" containerID="29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2"
Nov 01 09:06:07.605701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.602876    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2"} err="failed to get container status \"29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2\": rpc error: code = NotFound desc = could not find container \"29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2\": container with ID starting with 29149c1c0dbb03839eb2749f841719a4da2d2dfc907b1aaff5da13c96b9ef2a2 not found: ID does not exist"
Nov 01 09:06:07.665372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.665338    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbbrz\" (UniqueName: \"kubernetes.io/projected/af666f42-b43a-480e-a095-2f162a34be3e-kube-api-access-cbbrz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:07.665372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.665367    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af666f42-b43a-480e-a095-2f162a34be3e-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:07.750325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.750294    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"]
Nov 01 09:06:07.756348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:07.756318    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-fd77bc899-d87wf"]
Nov 01 09:06:08.929444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:08.929409    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af666f42-b43a-480e-a095-2f162a34be3e" path="/var/lib/kubelet/pods/af666f42-b43a-480e-a095-2f162a34be3e/volumes"
Nov 01 09:06:09.030772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:09.030730    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:09.555095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:09.555067    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:09.700348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:09.700321    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:09.847135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:09.847066    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:10.969594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:10.969554    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:11.323386 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:06:11.323344    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:06:11.528854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:11.528808    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:15.889830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:15.889790    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:17.220611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:17.220571    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:17.374954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:17.374920    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:18.802075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:18.802042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:19.326377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:19.326343    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"]
Nov 01 09:06:19.326605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:19.326581    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy" containerID="cri-o://08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f" gracePeriod=30
Nov 01 09:06:19.343147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:19.343116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:20.157402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:20.157367    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:20.955691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:20.955655    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:24.578575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.578519    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:06:24.656912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.656853    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.656912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.656891    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.656924    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.656959    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.656979    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657050    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657100    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657144    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657234    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket\") pod \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\" (UID: \"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28\") "
Nov 01 09:06:24.657315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657250    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:06:24.657572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657463    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:06:24.657644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657578    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:06:24.657709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657685    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:06:24.657802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657781    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:06:24.657905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657890    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.657969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657906    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.657969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657916    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.657969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657927    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.657969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.657935    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.659182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.659161    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:06:24.659290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.659226    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "kube-api-access-gqwn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:06:24.659413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.659391    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:06:24.659509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.659412    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token" pod "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" (UID: "1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:06:24.695993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.695968    4314 generic.go:356] "Generic (PLEG): container finished" podID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerID="08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f" exitCode=0
Nov 01 09:06:24.696116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.696022    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"
Nov 01 09:06:24.696116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.696054    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" event={"ID":"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28","Type":"ContainerDied","Data":"08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f"}
Nov 01 09:06:24.696116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.696103    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp" event={"ID":"1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28","Type":"ContainerDied","Data":"98016881660ec11151c8d185654e1b62c7eb8ba34116ce11023495c143364a65"}
Nov 01 09:06:24.696224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.696126    4314 scope.go:122] "RemoveContainer" containerID="08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f"
Nov 01 09:06:24.706441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.706425    4314 scope.go:122] "RemoveContainer" containerID="08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f"
Nov 01 09:06:24.706724 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:06:24.706705    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f\": container with ID starting with 08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f not found: ID does not exist" containerID="08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f"
Nov 01 09:06:24.706792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.706736    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f"} err="failed to get container status \"08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f\": rpc error: code = NotFound desc = could not find container \"08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f\": container with ID starting with 08bbc6fcd606bee2a3e942910f3f11f82c554a22d4529d188878e37c03ce006f not found: ID does not exist"
Nov 01 09:06:24.759139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.759105    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqwn8\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-kube-api-access-gqwn8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.759139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.759140    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.759286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.759155    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.759286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.759169    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:24.846873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.846837    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"]
Nov 01 09:06:24.850653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.850626    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-45dhp"]
Nov 01 09:06:24.929040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:24.928975    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" path="/var/lib/kubelet/pods/1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28/volumes"
Nov 01 09:06:26.319813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:26.319779    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:26.472987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:26.472956    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:30.364747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:30.364712    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:30.897342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:30.897308    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:31.203569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:31.203463    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"]
Nov 01 09:06:31.330819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:31.330785    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:31.475409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:31.475328    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:31.625214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:31.625186    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:32.700816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.700778    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"]
Nov 01 09:06:32.701569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.701513    4314 cpu_manager.go:414] "containerMap: removing container" podUID="af666f42-b43a-480e-a095-2f162a34be3e" containerName="payload-pre-processing"
Nov 01 09:06:32.701569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.701564    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af666f42-b43a-480e-a095-2f162a34be3e" containerName="payload-pre-processing"
Nov 01 09:06:32.701783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.701605    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy"
Nov 01 09:06:32.701783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.701612    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy"
Nov 01 09:06:32.701783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.701734    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6ac689-a6a8-4ae6-a1eb-c7055b0b1d28" containerName="istio-proxy"
Nov 01 09:06:32.701783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.701757    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="af666f42-b43a-480e-a095-2f162a34be3e" containerName="payload-pre-processing"
Nov 01 09:06:32.706419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.706393    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:32.774480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.774449    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"]
Nov 01 09:06:32.817891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.817860    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-23fa20-dockercfg-kgwx9\""
Nov 01 09:06:32.818226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.818180    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-23fa20\""
Nov 01 09:06:32.855187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.855132    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftst\" (UniqueName: \"kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst\") pod \"payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:32.855399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.855221    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:32.956975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.956893    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftst\" (UniqueName: \"kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst\") pod \"payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:32.956975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.956933    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:32.957674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:32.957648    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:33.014411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.014385    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftst\" (UniqueName: \"kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst\") pod \"payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:33.023241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.023222    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:33.391924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.391899    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"]
Nov 01 09:06:33.393271 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:06:33.393242    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4597fbb_790a_4ee0_9bb0_5ac671cd2ae0.slice/crio-cab3ab8b97e8515db2f3d7f48e7a890b25076ab8a7109e39221488635cda97e4 WatchSource:0}: Error finding container cab3ab8b97e8515db2f3d7f48e7a890b25076ab8a7109e39221488635cda97e4: Status 404 returned error can't find the container with id cab3ab8b97e8515db2f3d7f48e7a890b25076ab8a7109e39221488635cda97e4
Nov 01 09:06:33.631691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.631663    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:33.753383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.753347    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h" event={"ID":"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0","Type":"ContainerStarted","Data":"735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2"}
Nov 01 09:06:33.753798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.753400    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h" event={"ID":"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0","Type":"ContainerStarted","Data":"cab3ab8b97e8515db2f3d7f48e7a890b25076ab8a7109e39221488635cda97e4"}
Nov 01 09:06:33.753798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.753453    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:33.864770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:33.864712    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h" podStartSLOduration=1.864693954 podStartE2EDuration="1.864693954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:33.861676096 +0000 UTC m=+2067.056070651" watchObservedRunningTime="2026-11-01 09:06:33.864693954 +0000 UTC m=+2067.059088511"
Nov 01 09:06:36.204669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.204630    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx" podUID="8071338b-a7a1-4741-810f-4c7d90400e3d" containerName="payload-pre-processing" containerID="cri-o://9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28" gracePeriod=25
Nov 01 09:06:36.465645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.465589    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:06:36.493570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.493525    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume\") pod \"8071338b-a7a1-4741-810f-4c7d90400e3d\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") "
Nov 01 09:06:36.493766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.493746    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp\" (UniqueName: \"kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp\") pod \"8071338b-a7a1-4741-810f-4c7d90400e3d\" (UID: \"8071338b-a7a1-4741-810f-4c7d90400e3d\") "
Nov 01 09:06:36.493828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.493803    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume" pod "8071338b-a7a1-4741-810f-4c7d90400e3d" (UID: "8071338b-a7a1-4741-810f-4c7d90400e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:06:36.494312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.494290    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8071338b-a7a1-4741-810f-4c7d90400e3d-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:36.495953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.495932    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp" pod "8071338b-a7a1-4741-810f-4c7d90400e3d" (UID: "8071338b-a7a1-4741-810f-4c7d90400e3d"). InnerVolumeSpecName "kube-api-access-q9nkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:06:36.595904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.595870    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nkp\" (UniqueName: \"kubernetes.io/projected/8071338b-a7a1-4741-810f-4c7d90400e3d-kube-api-access-q9nkp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:06:36.773107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.773076    4314 generic.go:356] "Generic (PLEG): container finished" podID="8071338b-a7a1-4741-810f-4c7d90400e3d" containerID="9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28" exitCode=0
Nov 01 09:06:36.773284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.773118    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"
Nov 01 09:06:36.773284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.773146    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx" event={"ID":"8071338b-a7a1-4741-810f-4c7d90400e3d","Type":"ContainerDied","Data":"9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28"}
Nov 01 09:06:36.773284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.773186    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx" event={"ID":"8071338b-a7a1-4741-810f-4c7d90400e3d","Type":"ContainerDied","Data":"3a79498802dc0d95b655b19025a75fe90d7eef9fbe5679a1b384b1d132f605ae"}
Nov 01 09:06:36.773284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.773207    4314 scope.go:122] "RemoveContainer" containerID="9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28"
Nov 01 09:06:36.782940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.782923    4314 scope.go:122] "RemoveContainer" containerID="9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28"
Nov 01 09:06:36.783199 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:06:36.783180    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28\": container with ID starting with 9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28 not found: ID does not exist" containerID="9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28"
Nov 01 09:06:36.783244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.783207    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28"} err="failed to get container status \"9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28\": rpc error: code = NotFound desc = could not find container \"9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28\": container with ID starting with 9a01f260f45466cc5f0bffd70afe4de282afdce0359ac1891912038239a47f28 not found: ID does not exist"
Nov 01 09:06:36.929591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.929566    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"]
Nov 01 09:06:36.934560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:36.934527    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-75f67688dc-2qqxx"]
Nov 01 09:06:38.693862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:38.693823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:38.867853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:38.867812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:38.957462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:38.957387    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8071338b-a7a1-4741-810f-4c7d90400e3d" path="/var/lib/kubelet/pods/8071338b-a7a1-4741-810f-4c7d90400e3d/volumes"
Nov 01 09:06:39.012263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:39.012226    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:39.362120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:39.362079    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:40.285331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.285301    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:40.883970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.883939    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"]
Nov 01 09:06:40.884524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.884504    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8071338b-a7a1-4741-810f-4c7d90400e3d" containerName="payload-pre-processing"
Nov 01 09:06:40.884698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.884681    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8071338b-a7a1-4741-810f-4c7d90400e3d" containerName="payload-pre-processing"
Nov 01 09:06:40.884960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.884944    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8071338b-a7a1-4741-810f-4c7d90400e3d" containerName="payload-pre-processing"
Nov 01 09:06:40.890101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.890082    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.949724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.949694    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4be7cc-openshift-default-dockercfg-4pvsc\""
Nov 01 09:06:40.961211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961175    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfdq2\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961230    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961315    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961371    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961427    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961569    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961632    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961706    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.961785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.961742    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:40.990927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:40.990897    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"]
Nov 01 09:06:41.054563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.054509    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:41.062625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.062590    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.062827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.062642    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.062827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.062698    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.062827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.062768    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.062855    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.062991    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063021    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063027    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063086    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063219    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfdq2\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063414    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063446    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063413    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.063697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.063677    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.066364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.066338    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.068431 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.068407    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.137998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.137921    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.138369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.138348    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfdq2\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2\") pod \"e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") " pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.206044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.206009    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:41.336673 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:06:41.336629    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:06:41.497168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.497134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:41.646799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.646768    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"]
Nov 01 09:06:41.648299 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:06:41.648269    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf536bc1f_0a5f_4109_8296_ad10e2687187.slice/crio-5d992eb7db64395d6a8f5703b3ffcbc64248be1aeaf510c24fc324fe63599ea5 WatchSource:0}: Error finding container 5d992eb7db64395d6a8f5703b3ffcbc64248be1aeaf510c24fc324fe63599ea5: Status 404 returned error can't find the container with id 5d992eb7db64395d6a8f5703b3ffcbc64248be1aeaf510c24fc324fe63599ea5
Nov 01 09:06:41.650860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.650826    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:06:41.650951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.650903    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:06:41.651036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.650947    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:06:41.809171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.809068    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" event={"ID":"f536bc1f-0a5f-4109-8296-ad10e2687187","Type":"ContainerStarted","Data":"279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"}
Nov 01 09:06:41.809171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.809135    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" event={"ID":"f536bc1f-0a5f-4109-8296-ad10e2687187","Type":"ContainerStarted","Data":"5d992eb7db64395d6a8f5703b3ffcbc64248be1aeaf510c24fc324fe63599ea5"}
Nov 01 09:06:41.927881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:41.927829    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podStartSLOduration=1.92781213 podStartE2EDuration="1.92781213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:41.926713194 +0000 UTC m=+2075.121107752" watchObservedRunningTime="2026-11-01 09:06:41.92781213 +0000 UTC m=+2075.122206685"
Nov 01 09:06:42.206671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:42.206579    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:42.212023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:42.211998    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:42.814900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:42.814865    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:42.815945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:42.815921    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:06:43.016091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:43.016060    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:43.208334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:43.208229    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:44.762432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:44.762404    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:06:44.918470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:44.918437    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:45.221500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:45.221415    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:45.452464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:45.452428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:45.655807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:45.655777    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:48.950361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:48.950335    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:49.194693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:49.194658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:50.478237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:50.478202    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:52.865764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:52.865729    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:53.016814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:53.016783    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:06:55.787694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:06:55.787652    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:02.580555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:02.579564    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:06.185836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:06.185804    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:06.695857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:06.695829    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:07.973137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:07.973034    4314 generic.go:356] "Generic (PLEG): container finished" podID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerID="cf1fec34c259b82aace395edff508246620565fb56f26e84cbdface393ddc5f5" exitCode=0
Nov 01 09:07:07.973798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:07.973772    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" event={"ID":"258e5053-6b33-47e6-8cee-ba4f4f1a0983","Type":"ContainerDied","Data":"cf1fec34c259b82aace395edff508246620565fb56f26e84cbdface393ddc5f5"}
Nov 01 09:07:07.974430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:07.974407    4314 scope.go:122] "RemoveContainer" containerID="cf1fec34c259b82aace395edff508246620565fb56f26e84cbdface393ddc5f5"
Nov 01 09:07:07.976265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:07.976233    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:07.976330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:07.976313    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:07.976367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:07.976345    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:08.150596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:08.150565    4314 scope.go:122] "RemoveContainer" containerID="4532d55302bdd459ffe26f83b17a3dbb958361cc9e0b0d5d9de793be6a9baf61"
Nov 01 09:07:08.990838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:08.990804    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" event={"ID":"258e5053-6b33-47e6-8cee-ba4f4f1a0983","Type":"ContainerStarted","Data":"f4ee1093c3a3a683b1cd2f80f822d247cd337852c665ead137debbf4780c38fc"}
Nov 01 09:07:09.077555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:09.076721    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:07:09.077555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:09.077100    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 09:07:09.077555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:09.077165    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 09:07:09.283997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:09.283907    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:10.075651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:10.075614    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 09:07:10.076240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:10.076204    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 09:07:11.069562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:11.069221    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 09:07:11.069562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:11.069285    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 09:07:11.322463 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:07:11.322380    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:07:11.924964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:11.924932    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:12.069890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:12.069851    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 09:07:12.070173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:12.070145    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 09:07:13.068841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:13.068806    4314 patch_prober.go:29] interesting pod/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 09:07:13.069464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:13.069432    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 09:07:14.072424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:14.072392    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:07:14.072905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:14.072817    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:07:14.073791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:14.073772    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:07:14.101193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:14.101170    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:14.284967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:14.284933    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:14.432883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:14.432812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:19.074026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:19.073947    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:23.864415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:23.864380    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:28.927303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:28.927263    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:33.877005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:33.876970    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:37.453381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:37.453325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:37.933844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:37.933809    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:39.040495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.040459    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"]
Nov 01 09:07:39.043720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.043701    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.096841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.096804    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.097035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.096897    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64g6\" (UniqueName: \"kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.097473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.097453    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb28ec\""
Nov 01 09:07:39.104452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.104417    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb28ec-dockercfg-wtc4q\""
Nov 01 09:07:39.146965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.146928    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"]
Nov 01 09:07:39.198351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.198307    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.198575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.198423    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64g6\" (UniqueName: \"kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.199109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.199087    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.295871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.295789    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64g6\" (UniqueName: \"kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6\") pod \"payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.355677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.355639    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:39.722637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.722607    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"]
Nov 01 09:07:39.723737 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:07:39.723709    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7388594_9627_4835_a96a_9af2fdac3409.slice/crio-07381d9f208dba5365b6eb61fb6ea2bf5e627f68ea41ba348cb2c80e4d7e97c5 WatchSource:0}: Error finding container 07381d9f208dba5365b6eb61fb6ea2bf5e627f68ea41ba348cb2c80e4d7e97c5: Status 404 returned error can't find the container with id 07381d9f208dba5365b6eb61fb6ea2bf5e627f68ea41ba348cb2c80e4d7e97c5
Nov 01 09:07:39.831337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.831305    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"]
Nov 01 09:07:39.839957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.839932    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:39.899226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.899190    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"]
Nov 01 09:07:39.907994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.907287    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:39.907994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.907366    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6gk\" (UniqueName: \"kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk\") pod \"payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:39.938140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:39.938107    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:40.009164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.009129    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:40.009361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.009194    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6gk\" (UniqueName: \"kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk\") pod \"payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:40.009935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.009911    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:40.076595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.076509    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6gk\" (UniqueName: \"kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk\") pod \"payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:40.183557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.180743    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:40.193590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.193441    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9" event={"ID":"e7388594-9627-4835-a96a-9af2fdac3409","Type":"ContainerStarted","Data":"67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529"}
Nov 01 09:07:40.193590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.193526    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:40.193590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.193584    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9" event={"ID":"e7388594-9627-4835-a96a-9af2fdac3409","Type":"ContainerStarted","Data":"07381d9f208dba5365b6eb61fb6ea2bf5e627f68ea41ba348cb2c80e4d7e97c5"}
Nov 01 09:07:40.305435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.304734    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"]
Nov 01 09:07:40.305435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.304924    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h" podUID="e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" containerName="payload-processing" containerID="cri-o://735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2" gracePeriod=30
Nov 01 09:07:40.310034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.309320    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9" podStartSLOduration=2.309300864 podStartE2EDuration="2.309300864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:40.302604961 +0000 UTC m=+2133.496999532" watchObservedRunningTime="2026-11-01 09:07:40.309300864 +0000 UTC m=+2133.503695421"
Nov 01 09:07:40.485556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.463907    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:40.609166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.609054    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:40.639162 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:07:40.638762    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cfe3f2_9356_4969_a0aa_4f48456c6b0d.slice/crio-e56d7f3d5009776213252f924a74f171d802624980faab2d759067c2ea28d266 WatchSource:0}: Error finding container e56d7f3d5009776213252f924a74f171d802624980faab2d759067c2ea28d266: Status 404 returned error can't find the container with id e56d7f3d5009776213252f924a74f171d802624980faab2d759067c2ea28d266
Nov 01 09:07:40.640218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.640190    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"]
Nov 01 09:07:40.709466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.709439    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:07:40.831603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.827917    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume\") pod \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") "
Nov 01 09:07:40.831603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.828204    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst\" (UniqueName: \"kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst\") pod \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\" (UID: \"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0\") "
Nov 01 09:07:40.855649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.854983    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume" pod "e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" (UID: "e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:07:40.903638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.895874    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst" pod "e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" (UID: "e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0"). InnerVolumeSpecName "kube-api-access-7ftst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:07:40.934258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.934223    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftst\" (UniqueName: \"kubernetes.io/projected/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-kube-api-access-7ftst\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:40.934491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:40.934474    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:41.201965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.201876    4314 generic.go:356] "Generic (PLEG): container finished" podID="e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" containerID="735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2" exitCode=0
Nov 01 09:07:41.201965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.201940    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"
Nov 01 09:07:41.202587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.201962    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h" event={"ID":"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0","Type":"ContainerDied","Data":"735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2"}
Nov 01 09:07:41.202587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.202024    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h" event={"ID":"e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0","Type":"ContainerDied","Data":"cab3ab8b97e8515db2f3d7f48e7a890b25076ab8a7109e39221488635cda97e4"}
Nov 01 09:07:41.202587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.202054    4314 scope.go:122] "RemoveContainer" containerID="735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2"
Nov 01 09:07:41.204454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.204404    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" event={"ID":"02cfe3f2-9356-4969-a0aa-4f48456c6b0d","Type":"ContainerStarted","Data":"db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69"}
Nov 01 09:07:41.204588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.204458    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" event={"ID":"02cfe3f2-9356-4969-a0aa-4f48456c6b0d","Type":"ContainerStarted","Data":"e56d7f3d5009776213252f924a74f171d802624980faab2d759067c2ea28d266"}
Nov 01 09:07:41.214241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.214223    4314 scope.go:122] "RemoveContainer" containerID="735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2"
Nov 01 09:07:41.214525 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:07:41.214502    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2\": container with ID starting with 735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2 not found: ID does not exist" containerID="735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2"
Nov 01 09:07:41.214670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.214565    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2"} err="failed to get container status \"735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2\": rpc error: code = NotFound desc = could not find container \"735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2\": container with ID starting with 735a0809ce2f0a7a6b08b0fb15da792d6599dc28f360b860a4b4b224f944cfd2 not found: ID does not exist"
Nov 01 09:07:41.324550 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:07:41.324501    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:07:41.364563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.363393    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" podStartSLOduration=2.3633738429999998 podStartE2EDuration="2.363373843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:41.361951128 +0000 UTC m=+2134.556345707" watchObservedRunningTime="2026-11-01 09:07:41.363373843 +0000 UTC m=+2134.557768399"
Nov 01 09:07:41.499558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.495751    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"]
Nov 01 09:07:41.510163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:41.509762    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-23fa20-55464b8664-rk95h"]
Nov 01 09:07:42.135390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:42.135350    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:42.213867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:42.213840    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:42.930240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:42.930209    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" path="/var/lib/kubelet/pods/e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0/volumes"
Nov 01 09:07:44.717439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:44.717405    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"]
Nov 01 09:07:44.717914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:44.717658    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy" containerID="cri-o://17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd" gracePeriod=30
Nov 01 09:07:45.621944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:45.621905    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:46.111140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:46.111105    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:48.787710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:48.787671    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:49.258197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.258165    4314 generic.go:356] "Generic (PLEG): container finished" podID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerID="c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1" exitCode=0
Nov 01 09:07:49.258426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.258246    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" event={"ID":"0fef78d8-13c5-4b75-8a1f-f769fe874167","Type":"ContainerDied","Data":"c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"}
Nov 01 09:07:49.258705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.258685    4314 scope.go:122] "RemoveContainer" containerID="c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"
Nov 01 09:07:49.260180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.260148    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:49.260275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.260217    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:49.260275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.260248    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:49.571845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.571759    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:07:49.990646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:49.990620    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 09:07:50.182765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.182673    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.182765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.182720    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.182846    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.182956    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.182985    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183058    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183111    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:50.183173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183118    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183134    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:07:50.183173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183159    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183209    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs\") pod \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\" (UID: \"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080\") "
Nov 01 09:07:50.183515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183318    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:50.183515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183360    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:50.183745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183617    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:50.183804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183767    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.183804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183782    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.183804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183794    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.183974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183808    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.183974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.183823    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.185029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.184998    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:07:50.185340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.185316    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:07:50.185508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.185490    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:50.185508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.185498    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px" pod "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" (UID: "e0513ff0-437c-4ca8-9ec3-c0ee53bf9080"). InnerVolumeSpecName "kube-api-access-bg8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:07:50.265607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.265572    4314 generic.go:356] "Generic (PLEG): container finished" podID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerID="17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd" exitCode=0
Nov 01 09:07:50.265796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.265621    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"
Nov 01 09:07:50.265796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.265655    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" event={"ID":"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080","Type":"ContainerDied","Data":"17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd"}
Nov 01 09:07:50.265796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.265726    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9" event={"ID":"e0513ff0-437c-4ca8-9ec3-c0ee53bf9080","Type":"ContainerDied","Data":"60ac5af3be3bc15df1cf5baee01a9f76296c69f4e45f526d93cfb03309bb9529"}
Nov 01 09:07:50.265796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.265760    4314 scope.go:122] "RemoveContainer" containerID="17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd"
Nov 01 09:07:50.267837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.267814    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" event={"ID":"0fef78d8-13c5-4b75-8a1f-f769fe874167","Type":"ContainerStarted","Data":"d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116"}
Nov 01 09:07:50.276136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.276119    4314 scope.go:122] "RemoveContainer" containerID="17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd"
Nov 01 09:07:50.276459 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:07:50.276437    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd\": container with ID starting with 17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd not found: ID does not exist" containerID="17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd"
Nov 01 09:07:50.276504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.276469    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd"} err="failed to get container status \"17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd\": rpc error: code = NotFound desc = could not find container \"17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd\": container with ID starting with 17ddfbab1ab1003d2570bf97ac26119dde3699a4383f183498a41e1e646457dd not found: ID does not exist"
Nov 01 09:07:50.285186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.285165    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.285186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.285187    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg8px\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-kube-api-access-bg8px\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.285311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.285199    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.285311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.285208    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:50.290591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.290566    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:50.442613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.439963    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:50.539237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.539198    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"]
Nov 01 09:07:50.586006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.585964    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b7bl9"]
Nov 01 09:07:50.844067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.844033    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:07:50.845378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.845353    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 09:07:50.845512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.845405    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 09:07:50.929586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:50.929556    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" path="/var/lib/kubelet/pods/e0513ff0-437c-4ca8-9ec3-c0ee53bf9080/volumes"
Nov 01 09:07:51.206757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:51.206672    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:07:51.778558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:51.777693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:51.844996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:51.844958    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 09:07:51.845152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:51.845016    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 09:07:52.320594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:52.320558    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"]
Nov 01 09:07:52.321133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:52.320772    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy" containerID="cri-o://307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a" gracePeriod=30
Nov 01 09:07:52.326375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:52.326347    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:52.844529 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:52.844491    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 09:07:52.844748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:52.844593    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 09:07:53.222784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:53.222686    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:07:53.792886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:53.792856    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:53.844929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:53.844886    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 09:07:53.845315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:53.845275    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 09:07:53.937406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:53.937376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:54.084138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.084039    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:54.232662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.232621    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:54.394363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394281    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"]
Nov 01 09:07:54.394879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394864    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" containerName="payload-processing"
Nov 01 09:07:54.394922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394880    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" containerName="payload-processing"
Nov 01 09:07:54.394922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394897    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy"
Nov 01 09:07:54.394922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394902    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy"
Nov 01 09:07:54.395023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394985    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4597fbb-790a-4ee0-9bb0-5ac671cd2ae0" containerName="payload-processing"
Nov 01 09:07:54.395023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.394994    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0513ff0-437c-4ca8-9ec3-c0ee53bf9080" containerName="istio-proxy"
Nov 01 09:07:54.400826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.400797    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437055    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437126    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dznx\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437167    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437193    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437331    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437411    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437439    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.437489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.437480    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.498381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.498341    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-616efa-openshift-default-dockercfg-9ln6r\""
Nov 01 09:07:54.525291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.524902    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540148    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540192    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540217    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540249    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540335    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540386    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dznx\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540427    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540456    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.541702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.540642    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.542948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.542926    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.543691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.543654    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.545580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.544064    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.545735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.545007    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.546200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.546178    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.546741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.546717    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.547168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.547146    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"]
Nov 01 09:07:54.548486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.548213    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.606869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.606775    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.610031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.609976    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dznx\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx\") pod \"e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") " pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.716796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.716715    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:54.844722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.844658    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 09:07:54.845207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.844766    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 09:07:54.907747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:54.907712    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:55.184170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.177241    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"]
Nov 01 09:07:55.184170 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:07:55.184132    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d67fde0_b276_4274_a498_f18d3308336b.slice/crio-320a1ac87143c8a07b42b48bf09455ad43c7d2d1d3e15aca89f067c8a8d85eb7 WatchSource:0}: Error finding container 320a1ac87143c8a07b42b48bf09455ad43c7d2d1d3e15aca89f067c8a8d85eb7: Status 404 returned error can't find the container with id 320a1ac87143c8a07b42b48bf09455ad43c7d2d1d3e15aca89f067c8a8d85eb7
Nov 01 09:07:55.187348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.187305    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:55.187518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.187394    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:55.187518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.187428    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:55.321975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.321248    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" event={"ID":"6d67fde0-b276-4274-a498-f18d3308336b","Type":"ContainerStarted","Data":"653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"}
Nov 01 09:07:55.321975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.321301    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" event={"ID":"6d67fde0-b276-4274-a498-f18d3308336b","Type":"ContainerStarted","Data":"320a1ac87143c8a07b42b48bf09455ad43c7d2d1d3e15aca89f067c8a8d85eb7"}
Nov 01 09:07:55.438848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.438742    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podStartSLOduration=1.438725454 podStartE2EDuration="1.438725454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:55.437274376 +0000 UTC m=+2148.631668933" watchObservedRunningTime="2026-11-01 09:07:55.438725454 +0000 UTC m=+2148.633120008"
Nov 01 09:07:55.717747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.717655    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:55.844167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.844127    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 09:07:55.844385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:55.844185    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 09:07:56.120297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:56.120250    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:56.723066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:56.723039    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:56.848470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:56.848438    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:07:56.848778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:56.848753    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:07:56.849404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:56.849385    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:07:56.924270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:56.924229    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:57.335416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.335386    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:57.336761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.336737    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:07:57.596915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.596882    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"]
Nov 01 09:07:57.605809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.605783    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:07:57.695940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.695907    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"]
Nov 01 09:07:57.699016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.698990    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699028    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699098    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699126    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699149    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699173    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699198    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699242    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699283    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data\") pod \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\" (UID: \"9d60b6bf-33f2-4162-a2d7-13ace4e85faa\") "
Nov 01 09:07:57.699638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699405    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:57.699638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699596    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:57.699638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699614    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:57.699795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699774    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:57.699859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.699838    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:07:57.700031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.700007    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.700111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.700035    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.700111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.700053    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.700111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.700069    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.700111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.700082    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.701272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.701250    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:07:57.701459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.701442    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:07:57.701506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.701468    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "kube-api-access-zhpv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:07:57.701853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.701835    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo" pod "9d60b6bf-33f2-4162-a2d7-13ace4e85faa" (UID: "9d60b6bf-33f2-4162-a2d7-13ace4e85faa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:07:57.804101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.801607    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.804101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.801645    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhpv6\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-kube-api-access-zhpv6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.804101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.801663    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.804101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.801676    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60b6bf-33f2-4162-a2d7-13ace4e85faa-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:07:57.830789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.830760    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:57.981843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:57.981810    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:58.123930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.123899    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:58.342973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.342873    4314 generic.go:356] "Generic (PLEG): container finished" podID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerID="307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a" exitCode=0
Nov 01 09:07:58.342973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.342912    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" event={"ID":"9d60b6bf-33f2-4162-a2d7-13ace4e85faa","Type":"ContainerDied","Data":"307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a"}
Nov 01 09:07:58.343521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.342984    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"
Nov 01 09:07:58.343521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.342988    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl" event={"ID":"9d60b6bf-33f2-4162-a2d7-13ace4e85faa","Type":"ContainerDied","Data":"66ee66d174ac195d6c45cd50948e094e4a26fb759a723d27aea0ad4dd46ffd12"}
Nov 01 09:07:58.343521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.343002    4314 scope.go:122] "RemoveContainer" containerID="307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a"
Nov 01 09:07:58.351709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.351685    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"]
Nov 01 09:07:58.352471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.352449    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy"
Nov 01 09:07:58.352582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.352474    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy"
Nov 01 09:07:58.352733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.352715    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy"
Nov 01 09:07:58.355812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.355794    4314 scope.go:122] "RemoveContainer" containerID="a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"
Nov 01 09:07:58.357786 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.357771    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.365853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.365838    4314 scope.go:122] "RemoveContainer" containerID="307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a"
Nov 01 09:07:58.366103 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:07:58.366082    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a\": container with ID starting with 307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a not found: ID does not exist" containerID="307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a"
Nov 01 09:07:58.366223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.366107    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a"} err="failed to get container status \"307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a\": rpc error: code = NotFound desc = could not find container \"307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a\": container with ID starting with 307a4d00eb939bd667bcf3fa5f51194bacbff4bab05ffb977f08198f3271138a not found: ID does not exist"
Nov 01 09:07:58.366223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.366124    4314 scope.go:122] "RemoveContainer" containerID="a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"
Nov 01 09:07:58.366357 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:07:58.366331    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c\": container with ID starting with a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c not found: ID does not exist" containerID="a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"
Nov 01 09:07:58.366420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.366351    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c"} err="failed to get container status \"a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c\": rpc error: code = NotFound desc = could not find container \"a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c\": container with ID starting with a935862d478a1d22f9dbea2961656e4d8a7836bd5da7f84b56a5e2048e19d45c not found: ID does not exist"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462295    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462446    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462502    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462579    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrsfz\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462716    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462799    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462860    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.462941    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.463098    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.472557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.463443    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5fe64d-openshift-default-dockercfg-qcg5v\""
Nov 01 09:07:58.476060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.476032    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"]
Nov 01 09:07:58.532851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.531388    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:58.537038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.536592    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"]
Nov 01 09:07:58.564142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564105    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564153    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564177    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrsfz\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564231    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564287    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564326    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564357    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564426    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.565095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564476    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.565095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564616    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.565095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564647    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.565095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564867    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.565095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.564935    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.565345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.565153    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.567065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.567039    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.567442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.567409    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.581616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.581595    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-23fa20-openshift-default-5c497c9549-cqtgl"]
Nov 01 09:07:58.623565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.623479    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.623693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.623678    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrsfz\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz\") pod \"e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") " pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.672646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.672613    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:07:58.741610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.741573    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:58.897814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.897718    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:58.930113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:58.930077    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" path="/var/lib/kubelet/pods/9d60b6bf-33f2-4162-a2d7-13ace4e85faa/volumes"
Nov 01 09:07:59.062975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.062943    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:59.132852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.132822    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"]
Nov 01 09:07:59.133435 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:07:59.133413    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74835641_05d9_4ac7_a04b_7bb3767d267d.slice/crio-426f0dcf640ea5de7bfc6397df9f71c3861ddd5735be17a6a5406064218ed925 WatchSource:0}: Error finding container 426f0dcf640ea5de7bfc6397df9f71c3861ddd5735be17a6a5406064218ed925: Status 404 returned error can't find the container with id 426f0dcf640ea5de7bfc6397df9f71c3861ddd5735be17a6a5406064218ed925
Nov 01 09:07:59.136129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.136093    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:59.136201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.136186    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:59.136241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.136216    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:07:59.330125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.330086    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:59.360396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.359656    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" event={"ID":"74835641-05d9-4ac7-a04b-7bb3767d267d","Type":"ContainerStarted","Data":"3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"}
Nov 01 09:07:59.360396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.359716    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" event={"ID":"74835641-05d9-4ac7-a04b-7bb3767d267d","Type":"ContainerStarted","Data":"426f0dcf640ea5de7bfc6397df9f71c3861ddd5735be17a6a5406064218ed925"}
Nov 01 09:07:59.477647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.473526    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podStartSLOduration=1.473506743 podStartE2EDuration="1.473506743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:59.47249417 +0000 UTC m=+2152.666888726" watchObservedRunningTime="2026-11-01 09:07:59.473506743 +0000 UTC m=+2152.667901300"
Nov 01 09:07:59.485206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.485172    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:07:59.672868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:07:59.672838    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:08:00.679093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:00.679060    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:08:01.371098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:01.371067    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:08:01.372196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:01.372176    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:08:02.410278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.410241    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:02.597230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.597197    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz" podUID="f7fccc27-83f0-43cf-b3a5-511a4723a371" containerName="payload-processing" containerID="cri-o://85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023" gracePeriod=25
Nov 01 09:08:02.697186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.697090    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m" podUID="fb172518-dee4-4b68-a05d-1a4ad4671d18" containerName="payload-pre-processing" containerID="cri-o://eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995" gracePeriod=25
Nov 01 09:08:02.860683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.860654    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:08:02.932936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.932902    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn\" (UniqueName: \"kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn\") pod \"f7fccc27-83f0-43cf-b3a5-511a4723a371\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") "
Nov 01 09:08:02.933176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.933159    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume\") pod \"f7fccc27-83f0-43cf-b3a5-511a4723a371\" (UID: \"f7fccc27-83f0-43cf-b3a5-511a4723a371\") "
Nov 01 09:08:02.933648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.933619    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume" pod "f7fccc27-83f0-43cf-b3a5-511a4723a371" (UID: "f7fccc27-83f0-43cf-b3a5-511a4723a371"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:08:02.935209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.935181    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn" pod "f7fccc27-83f0-43cf-b3a5-511a4723a371" (UID: "f7fccc27-83f0-43cf-b3a5-511a4723a371"). InnerVolumeSpecName "kube-api-access-xf7fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:08:02.973408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:02.973379    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:08:03.034749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.034720    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume\") pod \"fb172518-dee4-4b68-a05d-1a4ad4671d18\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") "
Nov 01 09:08:03.034919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.034774    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp\" (UniqueName: \"kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp\") pod \"fb172518-dee4-4b68-a05d-1a4ad4671d18\" (UID: \"fb172518-dee4-4b68-a05d-1a4ad4671d18\") "
Nov 01 09:08:03.035162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.035138    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume" pod "fb172518-dee4-4b68-a05d-1a4ad4671d18" (UID: "fb172518-dee4-4b68-a05d-1a4ad4671d18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:08:03.035240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.035222    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb172518-dee4-4b68-a05d-1a4ad4671d18-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:03.035277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.035245    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fccc27-83f0-43cf-b3a5-511a4723a371-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:03.035277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.035254    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7fn\" (UniqueName: \"kubernetes.io/projected/f7fccc27-83f0-43cf-b3a5-511a4723a371-kube-api-access-xf7fn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:03.036867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.036847    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp" pod "fb172518-dee4-4b68-a05d-1a4ad4671d18" (UID: "fb172518-dee4-4b68-a05d-1a4ad4671d18"). InnerVolumeSpecName "kube-api-access-7wtlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:08:03.136859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.136826    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtlp\" (UniqueName: \"kubernetes.io/projected/fb172518-dee4-4b68-a05d-1a4ad4671d18-kube-api-access-7wtlp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:03.383219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.383182    4314 generic.go:356] "Generic (PLEG): container finished" podID="fb172518-dee4-4b68-a05d-1a4ad4671d18" containerID="eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995" exitCode=0
Nov 01 09:08:03.383406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.383233    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"
Nov 01 09:08:03.383406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.383269    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m" event={"ID":"fb172518-dee4-4b68-a05d-1a4ad4671d18","Type":"ContainerDied","Data":"eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995"}
Nov 01 09:08:03.383406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.383323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m" event={"ID":"fb172518-dee4-4b68-a05d-1a4ad4671d18","Type":"ContainerDied","Data":"840fba7fe45e615d5d346c6bff17deb4d7a4feae10fe32b5f40b3c2e92b22a3b"}
Nov 01 09:08:03.383406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.383345    4314 scope.go:122] "RemoveContainer" containerID="eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995"
Nov 01 09:08:03.384793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.384760    4314 generic.go:356] "Generic (PLEG): container finished" podID="f7fccc27-83f0-43cf-b3a5-511a4723a371" containerID="85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023" exitCode=0
Nov 01 09:08:03.384887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.384801    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"
Nov 01 09:08:03.384887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.384841    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz" event={"ID":"f7fccc27-83f0-43cf-b3a5-511a4723a371","Type":"ContainerDied","Data":"85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023"}
Nov 01 09:08:03.384981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.384897    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz" event={"ID":"f7fccc27-83f0-43cf-b3a5-511a4723a371","Type":"ContainerDied","Data":"fcbd3133205272cc34a1b3501e8f81ba5929b50524564e5016023eb10c2911eb"}
Nov 01 09:08:03.393871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.393851    4314 scope.go:122] "RemoveContainer" containerID="eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995"
Nov 01 09:08:03.394198 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:03.394174    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995\": container with ID starting with eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995 not found: ID does not exist" containerID="eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995"
Nov 01 09:08:03.394290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.394209    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995"} err="failed to get container status \"eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995\": rpc error: code = NotFound desc = could not find container \"eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995\": container with ID starting with eb0693262aec5ae07f48ca824ed492fe5bd19be08c075359bd70f0a941684995 not found: ID does not exist"
Nov 01 09:08:03.394290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.394233    4314 scope.go:122] "RemoveContainer" containerID="85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023"
Nov 01 09:08:03.402916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.402897    4314 scope.go:122] "RemoveContainer" containerID="85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023"
Nov 01 09:08:03.403202 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:03.403185    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023\": container with ID starting with 85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023 not found: ID does not exist" containerID="85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023"
Nov 01 09:08:03.403240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.403211    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023"} err="failed to get container status \"85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023\": rpc error: code = NotFound desc = could not find container \"85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023\": container with ID starting with 85c743252566e98fc9e84b22a002e5a39ac2b7ccc44253e327182605fb8c2023 not found: ID does not exist"
Nov 01 09:08:03.517469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.517435    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:03.539737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.539711    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"]
Nov 01 09:08:03.543436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.543413    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-59b7cd6c8c-hk9bz"]
Nov 01 09:08:03.682449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.682358    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"]
Nov 01 09:08:03.687004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.686967    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-d54bd6bbc-r7p4m"]
Nov 01 09:08:03.689013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.688990    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:03.926716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.926686    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"]
Nov 01 09:08:03.927190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927178    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb172518-dee4-4b68-a05d-1a4ad4671d18" containerName="payload-pre-processing"
Nov 01 09:08:03.927237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927192    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb172518-dee4-4b68-a05d-1a4ad4671d18" containerName="payload-pre-processing"
Nov 01 09:08:03.927237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927207    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f7fccc27-83f0-43cf-b3a5-511a4723a371" containerName="payload-processing"
Nov 01 09:08:03.927237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927212    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fccc27-83f0-43cf-b3a5-511a4723a371" containerName="payload-processing"
Nov 01 09:08:03.927340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927241    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy"
Nov 01 09:08:03.927340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927246    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy"
Nov 01 09:08:03.927340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927320    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb172518-dee4-4b68-a05d-1a4ad4671d18" containerName="payload-pre-processing"
Nov 01 09:08:03.927340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927328    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fccc27-83f0-43cf-b3a5-511a4723a371" containerName="payload-processing"
Nov 01 09:08:03.927340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.927336    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d60b6bf-33f2-4162-a2d7-13ace4e85faa" containerName="istio-proxy"
Nov 01 09:08:03.931884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.931863    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:03.953902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.953871    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:03.954111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.954082    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxw8\" (UniqueName: \"kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8\") pod \"payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:03.985571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:03.985527    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"]
Nov 01 09:08:04.030557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.029824    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-616efa\""
Nov 01 09:08:04.030557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.029934    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-616efa-dockercfg-8fk2n\""
Nov 01 09:08:04.062556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.057121    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxw8\" (UniqueName: \"kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8\") pod \"payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:04.062556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.057405    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:04.062556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.058858    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:04.090711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.090673    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"]
Nov 01 09:08:04.094926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.094889    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.115372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.115347    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxw8\" (UniqueName: \"kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8\") pod \"payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:04.147937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.147895    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"]
Nov 01 09:08:04.158900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.158869    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtjx\" (UniqueName: \"kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx\") pod \"payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.159043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.159007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.245097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.245062    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:04.259927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.259893    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtjx\" (UniqueName: \"kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx\") pod \"payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.260037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.260014    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.260955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.260931    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.317689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.317652    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtjx\" (UniqueName: \"kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx\") pod \"payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.334006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.333976    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:04.408774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.408740    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:04.503393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.503298    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:04.617169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.617140    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"]
Nov 01 09:08:04.619475 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:08:04.619448    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90578f8b_1e7c_45fe_a5a9_a722fb8a7436.slice/crio-67ae29e3e1fa91a7c52f17232e1bd37e24a60e31d6fc8a8fc20d6a7a7958a7e6 WatchSource:0}: Error finding container 67ae29e3e1fa91a7c52f17232e1bd37e24a60e31d6fc8a8fc20d6a7a7958a7e6: Status 404 returned error can't find the container with id 67ae29e3e1fa91a7c52f17232e1bd37e24a60e31d6fc8a8fc20d6a7a7958a7e6
Nov 01 09:08:04.700712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.697069    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:04.798166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.790364    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"]
Nov 01 09:08:04.930884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.930851    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fccc27-83f0-43cf-b3a5-511a4723a371" path="/var/lib/kubelet/pods/f7fccc27-83f0-43cf-b3a5-511a4723a371/volumes"
Nov 01 09:08:04.931301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:04.931287    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb172518-dee4-4b68-a05d-1a4ad4671d18" path="/var/lib/kubelet/pods/fb172518-dee4-4b68-a05d-1a4ad4671d18/volumes"
Nov 01 09:08:05.399433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.399396    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp" event={"ID":"806da6f1-d584-4505-b6c4-73a2a2d60fa5","Type":"ContainerStarted","Data":"67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d"}
Nov 01 09:08:05.399674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.399460    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp" event={"ID":"806da6f1-d584-4505-b6c4-73a2a2d60fa5","Type":"ContainerStarted","Data":"a5905a79ef945f7ccb7bbc631f0420dbbfec87eedb8da0971e1110e3cb18fe1e"}
Nov 01 09:08:05.399674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.399601    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:05.401139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.401118    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn" event={"ID":"90578f8b-1e7c-45fe-a5a9-a722fb8a7436","Type":"ContainerStarted","Data":"7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256"}
Nov 01 09:08:05.401246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.401156    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn" event={"ID":"90578f8b-1e7c-45fe-a5a9-a722fb8a7436","Type":"ContainerStarted","Data":"67ae29e3e1fa91a7c52f17232e1bd37e24a60e31d6fc8a8fc20d6a7a7958a7e6"}
Nov 01 09:08:05.401246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.401208    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:05.511601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.511511    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp" podStartSLOduration=1.511494367 podStartE2EDuration="1.511494367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:05.511050252 +0000 UTC m=+2158.705444808" watchObservedRunningTime="2026-11-01 09:08:05.511494367 +0000 UTC m=+2158.705888923"
Nov 01 09:08:05.641912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:05.641846    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn" podStartSLOduration=2.641824999 podStartE2EDuration="2.641824999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:05.640189872 +0000 UTC m=+2158.834584452" watchObservedRunningTime="2026-11-01 09:08:05.641824999 +0000 UTC m=+2158.836219555"
Nov 01 09:08:06.860329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:06.860292    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:08.077993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.077951    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:08.312136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.312091    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:08.487609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.487564    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"]
Nov 01 09:08:08.493590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.493527    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.530268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.530231    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkbx\" (UniqueName: \"kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx\") pod \"payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.530516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.530492    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.546405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.546368    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"]
Nov 01 09:08:08.589223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.587755    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5fe64d\""
Nov 01 09:08:08.589223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.587906    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5fe64d-dockercfg-pwr7w\""
Nov 01 09:08:08.648477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.648436    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkbx\" (UniqueName: \"kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx\") pod \"payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.659562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.651037    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.659562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.651981    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.711362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.711321    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkbx\" (UniqueName: \"kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx\") pod \"payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:08.807794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:08.807701    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:09.122424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.122331    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:09.180820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.180788    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"]
Nov 01 09:08:09.181929 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:08:09.181904    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fcbeee_f3d0_4055_951f_4295f6890b47.slice/crio-d8ddc5008a9333688b6a8c1008439be9f13b78676048eecbbf4cc55502a9cf93 WatchSource:0}: Error finding container d8ddc5008a9333688b6a8c1008439be9f13b78676048eecbbf4cc55502a9cf93: Status 404 returned error can't find the container with id d8ddc5008a9333688b6a8c1008439be9f13b78676048eecbbf4cc55502a9cf93
Nov 01 09:08:09.434603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.434480    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz" event={"ID":"f1fcbeee-f3d0-4055-951f-4295f6890b47","Type":"ContainerStarted","Data":"cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941"}
Nov 01 09:08:09.434603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.434585    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz" event={"ID":"f1fcbeee-f3d0-4055-951f-4295f6890b47","Type":"ContainerStarted","Data":"d8ddc5008a9333688b6a8c1008439be9f13b78676048eecbbf4cc55502a9cf93"}
Nov 01 09:08:09.435658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.435634    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:09.543312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.543278    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"]
Nov 01 09:08:09.543591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.543547    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb" podUID="47544b66-9c4e-4999-b3ff-0441163e0b93" containerName="payload-pre-processing" containerID="cri-o://171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1" gracePeriod=30
Nov 01 09:08:09.545805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.545765    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz" podStartSLOduration=1.54575283 podStartE2EDuration="1.54575283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:09.543475839 +0000 UTC m=+2162.737870394" watchObservedRunningTime="2026-11-01 09:08:09.54575283 +0000 UTC m=+2162.740147425"
Nov 01 09:08:09.645139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.645105    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:09.787685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.787645    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:09.814639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.814614    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:08:09.865309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.865276    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz\" (UniqueName: \"kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz\") pod \"47544b66-9c4e-4999-b3ff-0441163e0b93\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") "
Nov 01 09:08:09.865503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.865323    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume\") pod \"47544b66-9c4e-4999-b3ff-0441163e0b93\" (UID: \"47544b66-9c4e-4999-b3ff-0441163e0b93\") "
Nov 01 09:08:09.865747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.865726    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume" pod "47544b66-9c4e-4999-b3ff-0441163e0b93" (UID: "47544b66-9c4e-4999-b3ff-0441163e0b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:08:09.867504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.867477    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz" pod "47544b66-9c4e-4999-b3ff-0441163e0b93" (UID: "47544b66-9c4e-4999-b3ff-0441163e0b93"). InnerVolumeSpecName "kube-api-access-7rdcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:08:09.966799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.966694    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rdcz\" (UniqueName: \"kubernetes.io/projected/47544b66-9c4e-4999-b3ff-0441163e0b93-kube-api-access-7rdcz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:09.966799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:09.966727    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47544b66-9c4e-4999-b3ff-0441163e0b93-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:10.442613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.442576    4314 generic.go:356] "Generic (PLEG): container finished" podID="47544b66-9c4e-4999-b3ff-0441163e0b93" containerID="171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1" exitCode=0
Nov 01 09:08:10.443025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.442623    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"
Nov 01 09:08:10.443025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.442657    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb" event={"ID":"47544b66-9c4e-4999-b3ff-0441163e0b93","Type":"ContainerDied","Data":"171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1"}
Nov 01 09:08:10.443025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.442716    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb" event={"ID":"47544b66-9c4e-4999-b3ff-0441163e0b93","Type":"ContainerDied","Data":"cb70a8eb3c734a76e0f92a24d7784c5de1382b07ae35fde1f13c8ec0825c1ee5"}
Nov 01 09:08:10.443025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.442738    4314 scope.go:122] "RemoveContainer" containerID="171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1"
Nov 01 09:08:10.453488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.453461    4314 scope.go:122] "RemoveContainer" containerID="171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1"
Nov 01 09:08:10.453891 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:10.453864    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1\": container with ID starting with 171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1 not found: ID does not exist" containerID="171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1"
Nov 01 09:08:10.454013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.453895    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1"} err="failed to get container status \"171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1\": rpc error: code = NotFound desc = could not find container \"171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1\": container with ID starting with 171a36fdffd53faad59ab4cd7bb357c9d433b877961d432a5949f9ede0a1b2d1 not found: ID does not exist"
Nov 01 09:08:10.596675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.596634    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"]
Nov 01 09:08:10.642105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.642074    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-85694c564c-wncjb"]
Nov 01 09:08:10.724917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.724838    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:10.867758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.867721    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:10.928949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:10.928916    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47544b66-9c4e-4999-b3ff-0441163e0b93" path="/var/lib/kubelet/pods/47544b66-9c4e-4999-b3ff-0441163e0b93/volumes"
Nov 01 09:08:11.171688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:11.171653    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:11.323768 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:11.323732    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:08:12.233451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:12.233417    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:16.410626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:16.410595    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:08:16.411048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:16.410665    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:08:16.815214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:16.815180    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:16.955314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:16.955278    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:17.781554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:17.780155    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:18.387205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:18.387169    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:18.871599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:18.871565    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:19.648241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:19.648206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:21.453334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:21.453305    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:08:23.018016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.017987    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:23.543816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.543769    4314 generic.go:356] "Generic (PLEG): container finished" podID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerID="279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf" exitCode=0
Nov 01 09:08:23.544035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.543838    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" event={"ID":"f536bc1f-0a5f-4109-8296-ad10e2687187","Type":"ContainerDied","Data":"279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"}
Nov 01 09:08:23.544316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.544301    4314 scope.go:122] "RemoveContainer" containerID="279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"
Nov 01 09:08:23.545916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.545891    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:08:23.545985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.545960    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:08:23.546021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:23.545994    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:08:24.551959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:24.551924    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" event={"ID":"f536bc1f-0a5f-4109-8296-ad10e2687187","Type":"ContainerStarted","Data":"be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed"}
Nov 01 09:08:25.206166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:25.206128    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:08:25.207168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:25.207138    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 09:08:25.207287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:25.207208    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 09:08:25.779556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:25.779202    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:26.206914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:26.206814    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 09:08:26.206914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:26.206881    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 09:08:27.207296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:27.207245    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 09:08:27.207804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:27.207326    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 09:08:27.815086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:27.815041    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:08:28.207110 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:28.207030    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 09:08:28.207323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:28.207292    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 09:08:28.212741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:28.212715    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:29.206746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:29.206704    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 09:08:29.206974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:29.206766    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 09:08:30.211734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:30.211702    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:08:30.212812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:30.212787    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:08:33.312937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:33.312888    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:34.068565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:34.068504    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:34.458687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:34.458602    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:34.619183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:34.619071    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:38.369187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:38.369151    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:41.324013 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:41.323967    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:08:42.573139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:42.573099    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:42.930930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:42.930836    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"]
Nov 01 09:08:42.931111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:42.931038    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" podUID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" containerName="payload-processing" containerID="cri-o://db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69" gracePeriod=30
Nov 01 09:08:42.979367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:42.979333    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"]
Nov 01 09:08:42.979578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:42.979557    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9" podUID="e7388594-9627-4835-a96a-9af2fdac3409" containerName="payload-pre-processing" containerID="cri-o://67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529" gracePeriod=30
Nov 01 09:08:43.082412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.082371    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:43.232969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.232935    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:43.251598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.251567    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:08:43.271401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.271375    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:08:43.331892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.331854    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume\") pod \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") "
Nov 01 09:08:43.332066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.331909    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk\" (UniqueName: \"kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk\") pod \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\" (UID: \"02cfe3f2-9356-4969-a0aa-4f48456c6b0d\") "
Nov 01 09:08:43.332378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.332336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume" pod "02cfe3f2-9356-4969-a0aa-4f48456c6b0d" (UID: "02cfe3f2-9356-4969-a0aa-4f48456c6b0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:08:43.334094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.334073    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk" pod "02cfe3f2-9356-4969-a0aa-4f48456c6b0d" (UID: "02cfe3f2-9356-4969-a0aa-4f48456c6b0d"). InnerVolumeSpecName "kube-api-access-fq6gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:08:43.434617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.433770    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6\" (UniqueName: \"kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6\") pod \"e7388594-9627-4835-a96a-9af2fdac3409\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") "
Nov 01 09:08:43.434617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.433856    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume\") pod \"e7388594-9627-4835-a96a-9af2fdac3409\" (UID: \"e7388594-9627-4835-a96a-9af2fdac3409\") "
Nov 01 09:08:43.453581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.435038    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume" pod "e7388594-9627-4835-a96a-9af2fdac3409" (UID: "e7388594-9627-4835-a96a-9af2fdac3409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:08:43.453581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.435679    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7388594-9627-4835-a96a-9af2fdac3409-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:43.453581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.435708    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:43.453581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.435726    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6gk\" (UniqueName: \"kubernetes.io/projected/02cfe3f2-9356-4969-a0aa-4f48456c6b0d-kube-api-access-fq6gk\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:43.486753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.486717    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6" pod "e7388594-9627-4835-a96a-9af2fdac3409" (UID: "e7388594-9627-4835-a96a-9af2fdac3409"). InnerVolumeSpecName "kube-api-access-j64g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:08:43.536994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.536956    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64g6\" (UniqueName: \"kubernetes.io/projected/e7388594-9627-4835-a96a-9af2fdac3409-kube-api-access-j64g6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:08:43.703780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.703692    4314 generic.go:356] "Generic (PLEG): container finished" podID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" containerID="db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69" exitCode=0
Nov 01 09:08:43.703780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.703736    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"
Nov 01 09:08:43.704273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.703776    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" event={"ID":"02cfe3f2-9356-4969-a0aa-4f48456c6b0d","Type":"ContainerDied","Data":"db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69"}
Nov 01 09:08:43.704273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.703855    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" event={"ID":"02cfe3f2-9356-4969-a0aa-4f48456c6b0d","Type":"ContainerDied","Data":"e56d7f3d5009776213252f924a74f171d802624980faab2d759067c2ea28d266"}
Nov 01 09:08:43.704273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.703890    4314 scope.go:122] "RemoveContainer" containerID="db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69"
Nov 01 09:08:43.705327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.705305    4314 generic.go:356] "Generic (PLEG): container finished" podID="e7388594-9627-4835-a96a-9af2fdac3409" containerID="67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529" exitCode=0
Nov 01 09:08:43.705416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.705356    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"
Nov 01 09:08:43.705416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.705357    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9" event={"ID":"e7388594-9627-4835-a96a-9af2fdac3409","Type":"ContainerDied","Data":"67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529"}
Nov 01 09:08:43.705416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.705402    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9" event={"ID":"e7388594-9627-4835-a96a-9af2fdac3409","Type":"ContainerDied","Data":"07381d9f208dba5365b6eb61fb6ea2bf5e627f68ea41ba348cb2c80e4d7e97c5"}
Nov 01 09:08:43.716880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.716851    4314 scope.go:122] "RemoveContainer" containerID="db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69"
Nov 01 09:08:43.717241 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:43.717217    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69\": container with ID starting with db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69 not found: ID does not exist" containerID="db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69"
Nov 01 09:08:43.717329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.717255    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69"} err="failed to get container status \"db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69\": rpc error: code = NotFound desc = could not find container \"db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69\": container with ID starting with db4a51932812831a29f85e1b0ea039b2b34674e6055dde3f71cdb2f1c3da0b69 not found: ID does not exist"
Nov 01 09:08:43.717329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.717281    4314 scope.go:122] "RemoveContainer" containerID="67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529"
Nov 01 09:08:43.726379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.726358    4314 scope.go:122] "RemoveContainer" containerID="67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529"
Nov 01 09:08:43.726683 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:08:43.726664    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529\": container with ID starting with 67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529 not found: ID does not exist" containerID="67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529"
Nov 01 09:08:43.726748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.726692    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529"} err="failed to get container status \"67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529\": rpc error: code = NotFound desc = could not find container \"67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529\": container with ID starting with 67319080ecbbd652c057b34255c8efadf48f9bbfffdc3bbad0aae72e7ffae529 not found: ID does not exist"
Nov 01 09:08:43.858208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.858176    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"]
Nov 01 09:08:43.903896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.903860    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84"]
Nov 01 09:08:43.965226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.965153    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"]
Nov 01 09:08:43.969931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:43.969905    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb28ec-6c8fd9d859-2l8b9"]
Nov 01 09:08:44.221454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:44.221359    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-7685fc49db-nxh84" podUID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.152:9004: i/o timeout"
Nov 01 09:08:44.583312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:44.583281    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:44.928953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:44.928863    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" path="/var/lib/kubelet/pods/02cfe3f2-9356-4969-a0aa-4f48456c6b0d/volumes"
Nov 01 09:08:44.929502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:44.929481    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7388594-9627-4835-a96a-9af2fdac3409" path="/var/lib/kubelet/pods/e7388594-9627-4835-a96a-9af2fdac3409/volumes"
Nov 01 09:08:49.147681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:49.147594    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:49.298730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:49.298688    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:50.865244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:50.865208    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:51.415404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:51.415370    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:51.573568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:51.571041    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:54.105438 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:54.105359    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:54.732426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:54.732364    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:54.966127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:54.966079    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:55.391623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:55.391585    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:55.882699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:55.882667    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:57.361455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:57.361420    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:58.540349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:58.540273    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:59.108274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:59.108239    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:59.270755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:59.270714    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:08:59.433251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:08:59.433004    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:02.248006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:02.247970    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:02.407494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:02.407455    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:03.717815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:03.717779    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:03.873968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:03.873937    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:04.025638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:04.025595    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:08.139631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:08.139584    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:09.139597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:09.139556    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:10.192591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:10.192559    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:10.341499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:10.341465    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:11.323112 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:09:11.323072    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:09:12.591978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:12.591944    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:13.872337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:13.872298    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:14.053431 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:14.053398    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:14.271105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:14.271064    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:16.829344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:16.829312    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:18.874057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:18.874018    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:23.970340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:23.970250    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:30.395951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:30.395909    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:32.046070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:32.046033    4314 generic.go:356] "Generic (PLEG): container finished" podID="6d67fde0-b276-4274-a498-f18d3308336b" containerID="653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548" exitCode=0
Nov 01 09:09:32.046675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:32.046112    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" event={"ID":"6d67fde0-b276-4274-a498-f18d3308336b","Type":"ContainerDied","Data":"653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"}
Nov 01 09:09:32.046755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:32.046724    4314 scope.go:122] "RemoveContainer" containerID="653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"
Nov 01 09:09:32.048287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:32.048250    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:09:32.048359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:32.048339    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:09:32.048395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:32.048373    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:09:33.053699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:33.053661    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" event={"ID":"6d67fde0-b276-4274-a498-f18d3308336b","Type":"ContainerStarted","Data":"46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc"}
Nov 01 09:09:33.717382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:33.717343    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:09:33.717661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:33.717638    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 09:09:33.717717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:33.717688    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 09:09:34.717411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:34.717370    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 09:09:34.717849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:34.717442    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 09:09:35.444741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:35.444709    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:35.717709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:35.717624    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 09:09:35.717709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:35.717687    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 09:09:36.718127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:36.718090    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 09:09:36.718692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:36.718150    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 09:09:37.717729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:37.717694    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 09:09:37.717940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:37.717758    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 09:09:38.722922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:38.722887    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:09:38.723429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:38.723379    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:09:38.724226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:38.724202    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:09:39.045230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:39.045193    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:39.512688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:39.512655    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:40.610105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610068    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"]
Nov 01 09:09:40.610679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610661    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e7388594-9627-4835-a96a-9af2fdac3409" containerName="payload-pre-processing"
Nov 01 09:09:40.610733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610680    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7388594-9627-4835-a96a-9af2fdac3409" containerName="payload-pre-processing"
Nov 01 09:09:40.610733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610707    4314 cpu_manager.go:414] "containerMap: removing container" podUID="47544b66-9c4e-4999-b3ff-0441163e0b93" containerName="payload-pre-processing"
Nov 01 09:09:40.610733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610712    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47544b66-9c4e-4999-b3ff-0441163e0b93" containerName="payload-pre-processing"
Nov 01 09:09:40.610834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610738    4314 cpu_manager.go:414] "containerMap: removing container" podUID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" containerName="payload-processing"
Nov 01 09:09:40.610834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610743    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" containerName="payload-processing"
Nov 01 09:09:40.610834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610826    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7388594-9627-4835-a96a-9af2fdac3409" containerName="payload-pre-processing"
Nov 01 09:09:40.610934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610839    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="47544b66-9c4e-4999-b3ff-0441163e0b93" containerName="payload-pre-processing"
Nov 01 09:09:40.610934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.610850    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cfe3f2-9356-4969-a0aa-4f48456c6b0d" containerName="payload-processing"
Nov 01 09:09:40.613255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.613235    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.708855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.708809    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0460d\""
Nov 01 09:09:40.709070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.709053    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0460d-dockercfg-dw459\""
Nov 01 09:09:40.712707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.712680    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"]
Nov 01 09:09:40.743368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.743328    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.743624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.743602    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxn8\" (UniqueName: \"kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.845207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.845172    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.845418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.845358    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxn8\" (UniqueName: \"kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.845917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.845892    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.902948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.902871    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxn8\" (UniqueName: \"kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:40.928873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:40.928844    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:41.295712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.295675    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"]
Nov 01 09:09:41.297430 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:09:41.297403    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7759674c_519c_4848_bbed_dc111a360e87.slice/crio-46f2fca3b01a1f1de69dbc8964c6c0b91b5f70f291946aaf83cfab15d72d7631 WatchSource:0}: Error finding container 46f2fca3b01a1f1de69dbc8964c6c0b91b5f70f291946aaf83cfab15d72d7631: Status 404 returned error can't find the container with id 46f2fca3b01a1f1de69dbc8964c6c0b91b5f70f291946aaf83cfab15d72d7631
Nov 01 09:09:41.329397 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:09:41.329352    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:09:41.409794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.409757    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"]
Nov 01 09:09:41.412574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.412524    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.468826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.468797    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"]
Nov 01 09:09:41.552956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.552865    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm25f\" (UniqueName: \"kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f\") pod \"payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.553118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.553078    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.653953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.653918    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.654452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.654032    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm25f\" (UniqueName: \"kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f\") pod \"payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.654659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.654638    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.711778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.711745    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm25f\" (UniqueName: \"kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f\") pod \"payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:41.724771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:41.724741    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:42.093995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.093963    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"]
Nov 01 09:09:42.095275 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:09:42.095234    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c8e1ee_f730_42fb_a022_ff61a23a6a3e.slice/crio-c86e7ea3d431e2d580f86a70aac97d8fa5d2712e21774f614467c1dddcaaa255 WatchSource:0}: Error finding container c86e7ea3d431e2d580f86a70aac97d8fa5d2712e21774f614467c1dddcaaa255: Status 404 returned error can't find the container with id c86e7ea3d431e2d580f86a70aac97d8fa5d2712e21774f614467c1dddcaaa255
Nov 01 09:09:42.123489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.123463    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:42.163572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.157678    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk" event={"ID":"90c8e1ee-f730-42fb-a022-ff61a23a6a3e","Type":"ContainerStarted","Data":"c86e7ea3d431e2d580f86a70aac97d8fa5d2712e21774f614467c1dddcaaa255"}
Nov 01 09:09:42.163572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.162101    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm" event={"ID":"7759674c-519c-4848-bbed-dc111a360e87","Type":"ContainerStarted","Data":"7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739"}
Nov 01 09:09:42.163572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.162165    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm" event={"ID":"7759674c-519c-4848-bbed-dc111a360e87","Type":"ContainerStarted","Data":"46f2fca3b01a1f1de69dbc8964c6c0b91b5f70f291946aaf83cfab15d72d7631"}
Nov 01 09:09:42.163572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.163138    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:42.274412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:42.274355    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm" podStartSLOduration=2.274340445 podStartE2EDuration="2.274340445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:42.271003984 +0000 UTC m=+2255.465398539" watchObservedRunningTime="2026-11-01 09:09:42.274340445 +0000 UTC m=+2255.468735000"
Nov 01 09:09:43.169367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:43.169327    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk" event={"ID":"90c8e1ee-f730-42fb-a022-ff61a23a6a3e","Type":"ContainerStarted","Data":"d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262"}
Nov 01 09:09:43.169819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:43.169585    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:43.279816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:43.279761    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk" podStartSLOduration=2.279744379 podStartE2EDuration="2.279744379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:43.278191608 +0000 UTC m=+2256.472586165" watchObservedRunningTime="2026-11-01 09:09:43.279744379 +0000 UTC m=+2256.474138935"
Nov 01 09:09:46.934720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:46.934691    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:51.797022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:51.796964    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:54.187693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:54.187657    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:09:54.188155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:54.187739    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:09:56.777469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:56.777435    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:57.271092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:57.271059    4314 generic.go:356] "Generic (PLEG): container finished" podID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerID="3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f" exitCode=0
Nov 01 09:09:57.271283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:57.271146    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" event={"ID":"74835641-05d9-4ac7-a04b-7bb3767d267d","Type":"ContainerDied","Data":"3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"}
Nov 01 09:09:57.271609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:57.271595    4314 scope.go:122] "RemoveContainer" containerID="3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"
Nov 01 09:09:57.273160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:57.273131    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:09:57.273219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:57.273204    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:09:57.273264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:57.273240    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:09:58.278802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:58.278766    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" event={"ID":"74835641-05d9-4ac7-a04b-7bb3767d267d","Type":"ContainerStarted","Data":"67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52"}
Nov 01 09:09:58.482677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:58.482640    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:09:58.680948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:58.680852    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:09:58.682723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:58.682640    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 09:09:58.682723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:58.682702    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 09:09:59.679556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:59.679220    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 09:09:59.679556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:09:59.679281    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 09:10:00.677724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:00.677683    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 09:10:00.678059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:00.678028    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 09:10:01.382560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:01.377635    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:10:01.677053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:01.676960    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 09:10:01.677325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:01.677293    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 09:10:02.682944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:02.682908    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 09:10:02.683386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:02.682964    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 09:10:03.528912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:03.528874    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:03.679034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:03.678991    4314 patch_prober.go:29] interesting pod/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 09:10:03.679329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:03.679298    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 09:10:04.677754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:04.677721    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:10:04.678866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:04.678840    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:10:04.721461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:04.721422    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:08.718342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:08.718311    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:10.415793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:10.415758    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:10.564430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:10.564391    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:10.715930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:10.715832    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:11.324755 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:10:11.324714    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:10:12.944861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:12.944832    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:15.915480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:15.915445    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:19.679002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:19.678912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:20.221236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:20.221202    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:21.317035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.316993    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"]
Nov 01 09:10:21.322347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.322320    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.399496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.399467    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.399697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.399552    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nsm\" (UniqueName: \"kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.417442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.417410    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4be7cc\""
Nov 01 09:10:21.417645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.417472    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4be7cc-dockercfg-8fd6w\""
Nov 01 09:10:21.421460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.421436    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"]
Nov 01 09:10:21.500592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.500553    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.500745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.500634    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nsm\" (UniqueName: \"kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.501319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.501300    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.558403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.558366    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nsm\" (UniqueName: \"kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:21.636627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:21.636543    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:22.004594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:22.004565    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"]
Nov 01 09:10:22.005801 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:10:22.005774    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092a625d_0a4f_4790_95bb_c75ad7d0dc6f.slice/crio-dae69169823989350f8e9aa3a9cc4ff37b646a53121bb16b8a7aac6b3a05a08a WatchSource:0}: Error finding container dae69169823989350f8e9aa3a9cc4ff37b646a53121bb16b8a7aac6b3a05a08a: Status 404 returned error can't find the container with id dae69169823989350f8e9aa3a9cc4ff37b646a53121bb16b8a7aac6b3a05a08a
Nov 01 09:10:22.432654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:22.432558    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8" event={"ID":"092a625d-0a4f-4790-95bb-c75ad7d0dc6f","Type":"ContainerStarted","Data":"618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b"}
Nov 01 09:10:22.432654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:22.432614    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8" event={"ID":"092a625d-0a4f-4790-95bb-c75ad7d0dc6f","Type":"ContainerStarted","Data":"dae69169823989350f8e9aa3a9cc4ff37b646a53121bb16b8a7aac6b3a05a08a"}
Nov 01 09:10:22.432654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:22.432636    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:22.544013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:22.543959    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8" podStartSLOduration=1.5439433710000001 podStartE2EDuration="1.543943371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:10:22.54047003 +0000 UTC m=+2295.734864585" watchObservedRunningTime="2026-11-01 09:10:22.543943371 +0000 UTC m=+2295.738337925"
Nov 01 09:10:23.087663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:23.087625    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:28.378861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:28.378828    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:31.697001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:31.696970    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:32.946383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:32.946349    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:33.441389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:33.441360    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:10:33.459517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:33.459467    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:33.616488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:33.616451    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:33.764700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:33.764668    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:37.994108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:37.994072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:38.187699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:38.187666    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:38.347217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:38.347136    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:38.801512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:38.801470    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:41.256278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:41.256243    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:41.323666 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:10:41.323625    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:10:41.630965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:41.630875    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:41.795808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:41.795777    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:42.039688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.039654    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"]
Nov 01 09:10:42.043721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.043702    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.099158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.099126    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9f4e9b\""
Nov 01 09:10:42.168964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.168932    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9f4e9b-dockercfg-8k88n\""
Nov 01 09:10:42.173052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.173015    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"]
Nov 01 09:10:42.181572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.181492    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8t8r\" (UniqueName: \"kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r\") pod \"payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.181804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.181611    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.217253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.217220    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"]
Nov 01 09:10:42.221261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.221239    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.255871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.255843    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"]
Nov 01 09:10:42.283387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.283359    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8t8r\" (UniqueName: \"kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r\") pod \"payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.283832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.283433    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.284258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.284233    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.341214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.341151    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8t8r\" (UniqueName: \"kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r\") pod \"payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.357004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.356977    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:42.385211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.385168    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrr6\" (UniqueName: \"kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6\") pod \"payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.385344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.385309    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.427663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.424773    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:42.486496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.486457    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrr6\" (UniqueName: \"kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6\") pod \"payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.486689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.486513    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.487236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.487217    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.543619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.543590    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrr6\" (UniqueName: \"kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6\") pod \"payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:42.726221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.726190    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"]
Nov 01 09:10:42.726976 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:10:42.726951    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edc26dc_fd9b_46cf_b31d_eb3ec788a84b.slice/crio-c36f3b902eac431740be5cf14f6ac6a2fe6317911f75226b2807c16f41c663b7 WatchSource:0}: Error finding container c36f3b902eac431740be5cf14f6ac6a2fe6317911f75226b2807c16f41c663b7: Status 404 returned error can't find the container with id c36f3b902eac431740be5cf14f6ac6a2fe6317911f75226b2807c16f41c663b7
Nov 01 09:10:42.772951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.772916    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"]
Nov 01 09:10:42.835201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:42.835173    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:43.027610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.027576    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:43.187803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.181841    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:43.209566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.209513    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"]
Nov 01 09:10:43.211014 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:10:43.210975    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd06ef19_c1f9_49e0_9eda_0d4fccb1da2f.slice/crio-ec683a4f84e5e9b91727081e728a9ed9407e608723d3aa375f9e17b7a7b4957b WatchSource:0}: Error finding container ec683a4f84e5e9b91727081e728a9ed9407e608723d3aa375f9e17b7a7b4957b: Status 404 returned error can't find the container with id ec683a4f84e5e9b91727081e728a9ed9407e608723d3aa375f9e17b7a7b4957b
Nov 01 09:10:43.490288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.490258    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:43.647673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.647631    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp" event={"ID":"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b","Type":"ContainerStarted","Data":"e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a"}
Nov 01 09:10:43.647968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.647934    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp" event={"ID":"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b","Type":"ContainerStarted","Data":"c36f3b902eac431740be5cf14f6ac6a2fe6317911f75226b2807c16f41c663b7"}
Nov 01 09:10:43.648807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.648776    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:43.650446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.650418    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf" event={"ID":"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f","Type":"ContainerStarted","Data":"91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7"}
Nov 01 09:10:43.650580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.650460    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf" event={"ID":"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f","Type":"ContainerStarted","Data":"ec683a4f84e5e9b91727081e728a9ed9407e608723d3aa375f9e17b7a7b4957b"}
Nov 01 09:10:43.650770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.650750    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:43.759366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.759249    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp" podStartSLOduration=2.759231366 podStartE2EDuration="2.759231366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:10:43.758289042 +0000 UTC m=+2316.952683599" watchObservedRunningTime="2026-11-01 09:10:43.759231366 +0000 UTC m=+2316.953625923"
Nov 01 09:10:43.868387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:43.868312    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf" podStartSLOduration=1.868286553 podStartE2EDuration="1.868286553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:10:43.867846475 +0000 UTC m=+2317.062241033" watchObservedRunningTime="2026-11-01 09:10:43.868286553 +0000 UTC m=+2317.062681109"
Nov 01 09:10:44.213471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:44.213392    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:44.361739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:44.361704    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:45.467820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:45.467782    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:45.621111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:45.621078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:47.183210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:47.183166    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:47.625159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:47.625126    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:47.774078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:47.773958    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz" podUID="f1fcbeee-f3d0-4055-951f-4295f6890b47" containerName="payload-pre-processing" containerID="cri-o://cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941" gracePeriod=25
Nov 01 09:10:48.051682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.051655    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:10:48.068689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.068662    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx\" (UniqueName: \"kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx\") pod \"f1fcbeee-f3d0-4055-951f-4295f6890b47\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") "
Nov 01 09:10:48.068815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.068745    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume\") pod \"f1fcbeee-f3d0-4055-951f-4295f6890b47\" (UID: \"f1fcbeee-f3d0-4055-951f-4295f6890b47\") "
Nov 01 09:10:48.069137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.069100    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume" pod "f1fcbeee-f3d0-4055-951f-4295f6890b47" (UID: "f1fcbeee-f3d0-4055-951f-4295f6890b47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:10:48.070763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.070739    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx" pod "f1fcbeee-f3d0-4055-951f-4295f6890b47" (UID: "f1fcbeee-f3d0-4055-951f-4295f6890b47"). InnerVolumeSpecName "kube-api-access-cqkbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:10:48.170293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.170212    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkbx\" (UniqueName: \"kubernetes.io/projected/f1fcbeee-f3d0-4055-951f-4295f6890b47-kube-api-access-cqkbx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:10:48.170293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.170241    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fcbeee-f3d0-4055-951f-4295f6890b47-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:10:48.384982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.384941    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:48.533242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.533204    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:48.686433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.686401    4314 generic.go:356] "Generic (PLEG): container finished" podID="f1fcbeee-f3d0-4055-951f-4295f6890b47" containerID="cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941" exitCode=0
Nov 01 09:10:48.686652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.686448    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"
Nov 01 09:10:48.686652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.686457    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz" event={"ID":"f1fcbeee-f3d0-4055-951f-4295f6890b47","Type":"ContainerDied","Data":"cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941"}
Nov 01 09:10:48.686652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.686511    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz" event={"ID":"f1fcbeee-f3d0-4055-951f-4295f6890b47","Type":"ContainerDied","Data":"d8ddc5008a9333688b6a8c1008439be9f13b78676048eecbbf4cc55502a9cf93"}
Nov 01 09:10:48.686652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.686570    4314 scope.go:122] "RemoveContainer" containerID="cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941"
Nov 01 09:10:48.699363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.699344    4314 scope.go:122] "RemoveContainer" containerID="cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941"
Nov 01 09:10:48.699688 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:10:48.699665    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941\": container with ID starting with cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941 not found: ID does not exist" containerID="cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941"
Nov 01 09:10:48.699765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.699695    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941"} err="failed to get container status \"cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941\": rpc error: code = NotFound desc = could not find container \"cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941\": container with ID starting with cd82863f33489a3b74ddd238e5de7f541552a362c496e6b48abb95d8ced1d941 not found: ID does not exist"
Nov 01 09:10:48.840132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.840050    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"]
Nov 01 09:10:48.886589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.886556    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5fe64d-6b6596c45f-zfspz"]
Nov 01 09:10:48.930162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:48.930134    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fcbeee-f3d0-4055-951f-4295f6890b47" path="/var/lib/kubelet/pods/f1fcbeee-f3d0-4055-951f-4295f6890b47/volumes"
Nov 01 09:10:51.015862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:51.015789    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:54.659619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:54.659590    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:10:54.774465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:54.774436    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:10:55.665824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:55.665793    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:10:57.982602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:10:57.982564    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:03.164063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:03.164028    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:08.471109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:08.471059    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:11.324191 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:11:11.324151    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:11:13.267806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:13.267770    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:13.413723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:13.413677    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:19.230285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:19.230254    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:19.733662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:19.733631    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:19.878243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:19.878192    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:21.080478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:21.080444    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:21.593516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:21.593483    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:21.735091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:21.735051    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:23.148950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:23.148901    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:23.736976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:23.736938    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"]
Nov 01 09:11:23.737177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:23.737154    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy" containerID="cri-o://f4ee1093c3a3a683b1cd2f80f822d247cd337852c665ead137debbf4780c38fc" gracePeriod=30
Nov 01 09:11:23.744105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:23.744078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:26.419160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:26.419121    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:27.449777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:27.449739    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:28.511074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:28.511039    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:28.944814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:28.944548    4314 generic.go:356] "Generic (PLEG): container finished" podID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerID="f4ee1093c3a3a683b1cd2f80f822d247cd337852c665ead137debbf4780c38fc" exitCode=0
Nov 01 09:11:28.944814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:28.944588    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" event={"ID":"258e5053-6b33-47e6-8cee-ba4f4f1a0983","Type":"ContainerDied","Data":"f4ee1093c3a3a683b1cd2f80f822d247cd337852c665ead137debbf4780c38fc"}
Nov 01 09:11:28.944814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:28.944673    4314 scope.go:122] "RemoveContainer" containerID="cf1fec34c259b82aace395edff508246620565fb56f26e84cbdface393ddc5f5"
Nov 01 09:11:29.027903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.027840    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:11:29.083793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.083699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:29.198055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.198016    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.205554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.198334    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.216648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.216616    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.225202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.216907    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.225202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.223660    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.225202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.223722    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.225202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.223755    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.225202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.223828    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.225202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.223896    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert\") pod \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\" (UID: \"258e5053-6b33-47e6-8cee-ba4f4f1a0983\") "
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.209630    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.210209    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.216950    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.219633    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.224461    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.224922    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.227350    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:11:29.227815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.227738    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "kube-api-access-k8xwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:11:29.229221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.229202    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy" pod "258e5053-6b33-47e6-8cee-ba4f4f1a0983" (UID: "258e5053-6b33-47e6-8cee-ba4f4f1a0983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:29.229961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.229841    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xwp\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-kube-api-access-k8xwp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.229969    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.229986    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.230000    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.230025    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.230039    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.230052    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.230067    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5053-6b33-47e6-8cee-ba4f4f1a0983-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.230105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.230082    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5053-6b33-47e6-8cee-ba4f4f1a0983-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:29.242255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.242186    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:29.511104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.511066    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:29.952549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.952456    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"
Nov 01 09:11:29.952549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.952495    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87" event={"ID":"258e5053-6b33-47e6-8cee-ba4f4f1a0983","Type":"ContainerDied","Data":"2dbaf6dcf8114931735d9e5654714b706724f639ee63332f44903c35194533a6"}
Nov 01 09:11:29.952766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:29.952567    4314 scope.go:122] "RemoveContainer" containerID="f4ee1093c3a3a683b1cd2f80f822d247cd337852c665ead137debbf4780c38fc"
Nov 01 09:11:30.153280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:30.153249    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"]
Nov 01 09:11:30.199411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:30.199374    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-eaabf4-openshift-default-7f4b89f7bf-nqg87"]
Nov 01 09:11:30.928716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:30.928679    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" path="/var/lib/kubelet/pods/258e5053-6b33-47e6-8cee-ba4f4f1a0983/volumes"
Nov 01 09:11:33.198579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:33.198528    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:33.350999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:33.350940    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:33.499601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:33.499566    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:35.074352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:35.074310    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:36.286441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:36.286405    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:36.447620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:36.447591    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:38.817700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:38.817660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:41.323511 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:11:41.323469    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:11:43.429092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:43.429062    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:45.122005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:45.121965    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:45.282688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:45.282642    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:45.430224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:45.430115    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:50.183219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:50.183134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:50.335306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:50.335267    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:50.478594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:50.478473    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:51.842262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:51.842225    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:51.989395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:51.989179    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:52.486811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:52.486778    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"]
Nov 01 09:11:52.487025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:52.487001    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy" containerID="cri-o://67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52" gracePeriod=30
Nov 01 09:11:52.504265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:52.504234    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:53.954789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:53.954762    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:54.561054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:54.561015    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:54.799318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:54.799283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:55.461272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:55.461235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:56.899250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:56.899208    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:57.785390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.785358    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:11:57.805382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805352    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805409    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805434    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805467    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805574    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805611    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805635    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:57.805896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805709    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:57.805896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805776    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805833    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.805896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805840    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:57.805896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805889    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz\") pod \"74835641-05d9-4ac7-a04b-7bb3767d267d\" (UID: \"74835641-05d9-4ac7-a04b-7bb3767d267d\") "
Nov 01 09:11:57.806153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.806024    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:57.806153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.805952    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:11:57.806400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.806373    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.806400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.806400    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.806612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.806415    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.806612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.806431    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74835641-05d9-4ac7-a04b-7bb3767d267d-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.806612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.806448    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.807976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.807949    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:11:57.808071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.807955    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:11:57.808071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.808021    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:11:57.808071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.808042    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz" pod "74835641-05d9-4ac7-a04b-7bb3767d267d" (UID: "74835641-05d9-4ac7-a04b-7bb3767d267d"). InnerVolumeSpecName "kube-api-access-nrsfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:11:57.907851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.907809    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.907851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.907845    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrsfz\" (UniqueName: \"kubernetes.io/projected/74835641-05d9-4ac7-a04b-7bb3767d267d-kube-api-access-nrsfz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.907851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.907859    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:57.908338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:57.907868    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74835641-05d9-4ac7-a04b-7bb3767d267d-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:11:58.151159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.151066    4314 generic.go:356] "Generic (PLEG): container finished" podID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerID="67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52" exitCode=0
Nov 01 09:11:58.151159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.151124    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"
Nov 01 09:11:58.151390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.151156    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" event={"ID":"74835641-05d9-4ac7-a04b-7bb3767d267d","Type":"ContainerDied","Data":"67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52"}
Nov 01 09:11:58.151390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.151216    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576" event={"ID":"74835641-05d9-4ac7-a04b-7bb3767d267d","Type":"ContainerDied","Data":"426f0dcf640ea5de7bfc6397df9f71c3861ddd5735be17a6a5406064218ed925"}
Nov 01 09:11:58.151390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.151239    4314 scope.go:122] "RemoveContainer" containerID="67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52"
Nov 01 09:11:58.164641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.164616    4314 scope.go:122] "RemoveContainer" containerID="3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"
Nov 01 09:11:58.174578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.174557    4314 scope.go:122] "RemoveContainer" containerID="67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52"
Nov 01 09:11:58.174895 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:11:58.174872    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52\": container with ID starting with 67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52 not found: ID does not exist" containerID="67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52"
Nov 01 09:11:58.174942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.174906    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52"} err="failed to get container status \"67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52\": rpc error: code = NotFound desc = could not find container \"67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52\": container with ID starting with 67ec28bb0fcd71dee44f04028a3c2511580418f065da72fd461c6b9e6fb38b52 not found: ID does not exist"
Nov 01 09:11:58.174942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.174926    4314 scope.go:122] "RemoveContainer" containerID="3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"
Nov 01 09:11:58.175217 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:11:58.175196    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f\": container with ID starting with 3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f not found: ID does not exist" containerID="3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"
Nov 01 09:11:58.175273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.175223    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f"} err="failed to get container status \"3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f\": rpc error: code = NotFound desc = could not find container \"3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f\": container with ID starting with 3f7e0f5dd173cba1d077365f9ca5cef06a4222662d1dcff61b4639d277e74f2f not found: ID does not exist"
Nov 01 09:11:58.306334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.306294    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"]
Nov 01 09:11:58.310258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.310233    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5fe64d-openshift-default-544456f6df-q9576"]
Nov 01 09:11:58.322170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.322142    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:58.496664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.496628    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:58.642439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.642405    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:11:58.938722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:58.938640    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" path="/var/lib/kubelet/pods/74835641-05d9-4ac7-a04b-7bb3767d267d/volumes"
Nov 01 09:11:59.909123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:11:59.909089    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:00.379853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.379555    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"]
Nov 01 09:12:00.380408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.380063    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk" podUID="90c8e1ee-f730-42fb-a022-ff61a23a6a3e" containerName="payload-processing" containerID="cri-o://d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262" gracePeriod=30
Nov 01 09:12:00.432709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.432672    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"]
Nov 01 09:12:00.432902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.432886    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm" podUID="7759674c-519c-4848-bbed-dc111a360e87" containerName="payload-pre-processing" containerID="cri-o://7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739" gracePeriod=30
Nov 01 09:12:00.542781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.541096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:00.685072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.684912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:00.709258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.709228    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:12:00.742228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.742200    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:12:00.744399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.744378    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f\" (UniqueName: \"kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f\") pod \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") "
Nov 01 09:12:00.744559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.744546    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume\") pod \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\" (UID: \"90c8e1ee-f730-42fb-a022-ff61a23a6a3e\") "
Nov 01 09:12:00.744892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.744867    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume" pod "90c8e1ee-f730-42fb-a022-ff61a23a6a3e" (UID: "90c8e1ee-f730-42fb-a022-ff61a23a6a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:12:00.746432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.746412    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f" pod "90c8e1ee-f730-42fb-a022-ff61a23a6a3e" (UID: "90c8e1ee-f730-42fb-a022-ff61a23a6a3e"). InnerVolumeSpecName "kube-api-access-dm25f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:12:00.845707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.845673    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8\" (UniqueName: \"kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8\") pod \"7759674c-519c-4848-bbed-dc111a360e87\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") "
Nov 01 09:12:00.845913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.845770    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume\") pod \"7759674c-519c-4848-bbed-dc111a360e87\" (UID: \"7759674c-519c-4848-bbed-dc111a360e87\") "
Nov 01 09:12:00.846274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.846245    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume" pod "7759674c-519c-4848-bbed-dc111a360e87" (UID: "7759674c-519c-4848-bbed-dc111a360e87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:12:00.846398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.846388    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm25f\" (UniqueName: \"kubernetes.io/projected/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-kube-api-access-dm25f\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.846463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.846424    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8e1ee-f730-42fb-a022-ff61a23a6a3e-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.846463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.846434    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7759674c-519c-4848-bbed-dc111a360e87-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.847928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.847906    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8" pod "7759674c-519c-4848-bbed-dc111a360e87" (UID: "7759674c-519c-4848-bbed-dc111a360e87"). InnerVolumeSpecName "kube-api-access-nwxn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:12:00.948011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:00.947912    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxn8\" (UniqueName: \"kubernetes.io/projected/7759674c-519c-4848-bbed-dc111a360e87-kube-api-access-nwxn8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:01.172772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.172738    4314 generic.go:356] "Generic (PLEG): container finished" podID="90c8e1ee-f730-42fb-a022-ff61a23a6a3e" containerID="d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262" exitCode=0
Nov 01 09:12:01.172987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.172783    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"
Nov 01 09:12:01.172987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.172825    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk" event={"ID":"90c8e1ee-f730-42fb-a022-ff61a23a6a3e","Type":"ContainerDied","Data":"d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262"}
Nov 01 09:12:01.172987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.172886    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk" event={"ID":"90c8e1ee-f730-42fb-a022-ff61a23a6a3e","Type":"ContainerDied","Data":"c86e7ea3d431e2d580f86a70aac97d8fa5d2712e21774f614467c1dddcaaa255"}
Nov 01 09:12:01.172987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.172917    4314 scope.go:122] "RemoveContainer" containerID="d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262"
Nov 01 09:12:01.174180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.174157    4314 generic.go:356] "Generic (PLEG): container finished" podID="7759674c-519c-4848-bbed-dc111a360e87" containerID="7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739" exitCode=0
Nov 01 09:12:01.174259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.174197    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"
Nov 01 09:12:01.174259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.174216    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm" event={"ID":"7759674c-519c-4848-bbed-dc111a360e87","Type":"ContainerDied","Data":"7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739"}
Nov 01 09:12:01.174259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.174255    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm" event={"ID":"7759674c-519c-4848-bbed-dc111a360e87","Type":"ContainerDied","Data":"46f2fca3b01a1f1de69dbc8964c6c0b91b5f70f291946aaf83cfab15d72d7631"}
Nov 01 09:12:01.182717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.182695    4314 scope.go:122] "RemoveContainer" containerID="d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262"
Nov 01 09:12:01.183011 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:01.182992    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262\": container with ID starting with d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262 not found: ID does not exist" containerID="d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262"
Nov 01 09:12:01.183054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.183019    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262"} err="failed to get container status \"d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262\": rpc error: code = NotFound desc = could not find container \"d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262\": container with ID starting with d05ead5f2b3a1c6ca078e19ed42b895bf0fa9e80c40d4365fb43e69bd25c2262 not found: ID does not exist"
Nov 01 09:12:01.183054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.183036    4314 scope.go:122] "RemoveContainer" containerID="7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739"
Nov 01 09:12:01.192232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.192210    4314 scope.go:122] "RemoveContainer" containerID="7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739"
Nov 01 09:12:01.192589 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:01.192568    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739\": container with ID starting with 7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739 not found: ID does not exist" containerID="7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739"
Nov 01 09:12:01.192638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.192600    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739"} err="failed to get container status \"7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739\": rpc error: code = NotFound desc = could not find container \"7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739\": container with ID starting with 7dfef5c0313add63e0ca7940585f224388d00d71a7f7ca3bbf0e351b88981739 not found: ID does not exist"
Nov 01 09:12:01.327064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.326982    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"]
Nov 01 09:12:01.330331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.330302    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-578cf767ff-6phkm"]
Nov 01 09:12:01.435258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.435221    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"]
Nov 01 09:12:01.442095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.442065    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-57c7fd8bc4-xwvfk"]
Nov 01 09:12:01.457805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.457774    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:01.625747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.625645    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:01.868756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.868724    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"]
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869239    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f1fcbeee-f3d0-4055-951f-4295f6890b47" containerName="payload-pre-processing"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869258    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fcbeee-f3d0-4055-951f-4295f6890b47" containerName="payload-pre-processing"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869277    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7759674c-519c-4848-bbed-dc111a360e87" containerName="payload-pre-processing"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869285    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7759674c-519c-4848-bbed-dc111a360e87" containerName="payload-pre-processing"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869299    4314 cpu_manager.go:414] "containerMap: removing container" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869306    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869359    4314 cpu_manager.go:414] "containerMap: removing container" podUID="90c8e1ee-f730-42fb-a022-ff61a23a6a3e" containerName="payload-processing"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869367    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c8e1ee-f730-42fb-a022-ff61a23a6a3e" containerName="payload-processing"
Nov 01 09:12:01.869376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869383    4314 cpu_manager.go:414] "containerMap: removing container" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869412    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869426    4314 cpu_manager.go:414] "containerMap: removing container" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869432    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869573    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c8e1ee-f730-42fb-a022-ff61a23a6a3e" containerName="payload-processing"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869590    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869601    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="258e5053-6b33-47e6-8cee-ba4f4f1a0983" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869611    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869622    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fcbeee-f3d0-4055-951f-4295f6890b47" containerName="payload-pre-processing"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869633    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7759674c-519c-4848-bbed-dc111a360e87" containerName="payload-pre-processing"
Nov 01 09:12:01.869823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.869640    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy"
Nov 01 09:12:01.873137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.873118    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:01.923081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.922931    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-28efa1\""
Nov 01 09:12:01.923374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.923341    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-28efa1-dockercfg-kc6x7\""
Nov 01 09:12:01.926560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.926522    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"]
Nov 01 09:12:01.937139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.937110    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:01.966087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.964838    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzmj\" (UniqueName: \"kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj\") pod \"payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:01.966087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:01.964964    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:02.066360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.066319    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzmj\" (UniqueName: \"kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj\") pod \"payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:02.066586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.066403    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:02.067203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.067183    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:02.123192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.123162    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzmj\" (UniqueName: \"kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj\") pod \"payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:02.187356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.187263    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:02.557427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.557396    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"]
Nov 01 09:12:02.558693 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:12:02.558668    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c28f4b_375a_48c4_8e21_59d268ec866a.slice/crio-b252d7eddca5c551627c57c479905dae95272ffe97e3b817db8285e7c97a5f68 WatchSource:0}: Error finding container b252d7eddca5c551627c57c479905dae95272ffe97e3b817db8285e7c97a5f68: Status 404 returned error can't find the container with id b252d7eddca5c551627c57c479905dae95272ffe97e3b817db8285e7c97a5f68
Nov 01 09:12:02.929425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.929339    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7759674c-519c-4848-bbed-dc111a360e87" path="/var/lib/kubelet/pods/7759674c-519c-4848-bbed-dc111a360e87/volumes"
Nov 01 09:12:02.929843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:02.929827    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c8e1ee-f730-42fb-a022-ff61a23a6a3e" path="/var/lib/kubelet/pods/90c8e1ee-f730-42fb-a022-ff61a23a6a3e/volumes"
Nov 01 09:12:03.178361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:03.178329    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:03.190407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:03.190311    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm" event={"ID":"f1c28f4b-375a-48c4-8e21-59d268ec866a","Type":"ContainerStarted","Data":"264689f5c6c96f7042eea6c34d5357abef8ce8c53e35b99368585c53c3ae2e65"}
Nov 01 09:12:03.190407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:03.190378    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm" event={"ID":"f1c28f4b-375a-48c4-8e21-59d268ec866a","Type":"ContainerStarted","Data":"b252d7eddca5c551627c57c479905dae95272ffe97e3b817db8285e7c97a5f68"}
Nov 01 09:12:03.190644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:03.190504    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:03.298737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:03.298436    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm" podStartSLOduration=2.2984175589999998 podStartE2EDuration="2.298417559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:03.298068696 +0000 UTC m=+2396.492463253" watchObservedRunningTime="2026-11-01 09:12:03.298417559 +0000 UTC m=+2396.492812116"
Nov 01 09:12:03.318887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:03.318849    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:04.686775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:04.686725    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:05.176068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:05.176030    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"]
Nov 01 09:12:05.176278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:05.176252    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy" containerID="cri-o://d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116" gracePeriod=30
Nov 01 09:12:05.182703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:05.182664    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:06.634221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:06.634187    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:07.075433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.075398    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:07.123833 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.123798    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"]
Nov 01 09:12:07.124387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.124371    4314 cpu_manager.go:414] "containerMap: removing container" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy"
Nov 01 09:12:07.124434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.124387    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74835641-05d9-4ac7-a04b-7bb3767d267d" containerName="istio-proxy"
Nov 01 09:12:07.127409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.127386    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.176409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.176378    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6752e4-openshift-default-dockercfg-z6rpx\""
Nov 01 09:12:07.221475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.221381    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:07.228979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.228932    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.228988    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229035    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229135    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229193    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntcs\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229243    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229317    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229334    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"]
Nov 01 09:12:07.229351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229345    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.229631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.229403    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331545    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntcs\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331632    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331715    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331747    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331807    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331841    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.331922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331881    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.332219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.331925    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.332219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.332044    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.332522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.332458    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.332793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.332771    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.333387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.333355    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.333735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.333712    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.333893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.333861    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.336632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.336605    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.338357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.338327    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.397287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.397201    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.397287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.397202    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntcs\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.441173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.441133    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:07.458136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.458084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:07.867849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.867816    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"]
Nov 01 09:12:07.868876 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:12:07.868845    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6f7bd4_57ac_4fa6_9f1b_b5b7f010d07f.slice/crio-57ef1da670dc67d53dbe242cb3d78764a6275cec5d950af4bded01272867e804 WatchSource:0}: Error finding container 57ef1da670dc67d53dbe242cb3d78764a6275cec5d950af4bded01272867e804: Status 404 returned error can't find the container with id 57ef1da670dc67d53dbe242cb3d78764a6275cec5d950af4bded01272867e804
Nov 01 09:12:07.871599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.871519    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:12:07.871737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.871638    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:12:07.871737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:07.871670    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:12:08.228866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:08.228777    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" event={"ID":"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f","Type":"ContainerStarted","Data":"3bd022a8e6d6be4c87b232f62957af32e31ce44fc4e57509feae71ae0c7e49fe"}
Nov 01 09:12:08.228866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:08.228831    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" event={"ID":"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f","Type":"ContainerStarted","Data":"57ef1da670dc67d53dbe242cb3d78764a6275cec5d950af4bded01272867e804"}
Nov 01 09:12:08.345071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:08.345024    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podStartSLOduration=1.34500799 podStartE2EDuration="1.34500799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:08.34271858 +0000 UTC m=+2401.537113135" watchObservedRunningTime="2026-11-01 09:12:08.34500799 +0000 UTC m=+2401.539402604"
Nov 01 09:12:08.442070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:08.442031    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:08.448092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:08.448059    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:09.235513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:09.235485    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:09.236637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:09.236617    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:12:10.201903 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:10.201876    4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fef78d8_13c5_4b75_8a1f_f769fe874167.slice/crio-conmon-d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 09:12:10.445781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.445712    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:12:10.572932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.572892    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.572977    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573033    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573066    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573139    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573169    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573196    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573291    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:12:10.573474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573327    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:12:10.573474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573349    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573410    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs\") pod \"0fef78d8-13c5-4b75-8a1f-f769fe874167\" (UID: \"0fef78d8-13c5-4b75-8a1f-f769fe874167\") "
Nov 01 09:12:10.573474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573449    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:12:10.573843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573824    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:12:10.574038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573972    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.574038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.573999    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fef78d8-13c5-4b75-8a1f-f769fe874167-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.574038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.574013    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.574038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.574022    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.574384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.574354    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:12:10.575671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.575646    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "kube-api-access-xh7cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:12:10.575671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.575659    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:12:10.575912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.575889    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:12:10.575991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.575939    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy" pod "0fef78d8-13c5-4b75-8a1f-f769fe874167" (UID: "0fef78d8-13c5-4b75-8a1f-f769fe874167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:12:10.675700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.675665    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7cc\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-kube-api-access-xh7cc\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.675700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.675695    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.675700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.675706    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.675961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.675715    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fef78d8-13c5-4b75-8a1f-f769fe874167-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:10.675961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:10.675725    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fef78d8-13c5-4b75-8a1f-f769fe874167-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:11.060046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.059959    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:11.234541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.234505    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:11.249824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.249792    4314 generic.go:356] "Generic (PLEG): container finished" podID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerID="d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116" exitCode=0
Nov 01 09:12:11.250035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.249849    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"
Nov 01 09:12:11.250035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.249882    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" event={"ID":"0fef78d8-13c5-4b75-8a1f-f769fe874167","Type":"ContainerDied","Data":"d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116"}
Nov 01 09:12:11.250035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.249946    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h" event={"ID":"0fef78d8-13c5-4b75-8a1f-f769fe874167","Type":"ContainerDied","Data":"752eb59e42d8f2591f401466e77ae08ed6695b415d21eb5d963256884c273f6c"}
Nov 01 09:12:11.250035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.249976    4314 scope.go:122] "RemoveContainer" containerID="d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116"
Nov 01 09:12:11.260382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.260362    4314 scope.go:122] "RemoveContainer" containerID="c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"
Nov 01 09:12:11.269922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.269896    4314 scope.go:122] "RemoveContainer" containerID="d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116"
Nov 01 09:12:11.270258 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:11.270233    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116\": container with ID starting with d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116 not found: ID does not exist" containerID="d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116"
Nov 01 09:12:11.270332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.270274    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116"} err="failed to get container status \"d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116\": rpc error: code = NotFound desc = could not find container \"d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116\": container with ID starting with d68ef9edfdc696e92df47c35eb82e8ac0225b36d575ac97007cbb2f40dc45116 not found: ID does not exist"
Nov 01 09:12:11.270332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.270300    4314 scope.go:122] "RemoveContainer" containerID="c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"
Nov 01 09:12:11.270606 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:11.270573    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1\": container with ID starting with c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1 not found: ID does not exist" containerID="c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"
Nov 01 09:12:11.270688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.270607    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1"} err="failed to get container status \"c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1\": rpc error: code = NotFound desc = could not find container \"c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1\": container with ID starting with c078341529cc530491eb51f0451c8556a7ccc926205582e980dea9836b6b63d1 not found: ID does not exist"
Nov 01 09:12:11.325019 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:11.324981    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:12:11.392249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.392216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:11.418528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.418488    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"]
Nov 01 09:12:11.463263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:11.463223    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0460d-openshift-default-c54c55955-8wg5h"]
Nov 01 09:12:12.309817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:12.309786    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:12.929121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:12.929088    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" path="/var/lib/kubelet/pods/0fef78d8-13c5-4b75-8a1f-f769fe874167/volumes"
Nov 01 09:12:13.619391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:13.619358    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:14.199841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.199808    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:12:14.943625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.943586    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"]
Nov 01 09:12:14.944125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.944110    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy"
Nov 01 09:12:14.944178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.944126    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy"
Nov 01 09:12:14.944178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.944160    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy"
Nov 01 09:12:14.944178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.944165    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy"
Nov 01 09:12:14.944277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.944257    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy"
Nov 01 09:12:14.944277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.944270    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fef78d8-13c5-4b75-8a1f-f769fe874167" containerName="istio-proxy"
Nov 01 09:12:14.949175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.949150    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:14.951248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.951220    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h489\" (UniqueName: \"kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489\") pod \"payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:14.951430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:14.951254    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:15.000594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.000515    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"]
Nov 01 09:12:15.053119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.053082    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h489\" (UniqueName: \"kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489\") pod \"payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:15.053119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.053124    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:15.066694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.066658    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6752e4\""
Nov 01 09:12:15.066694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.066679    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6752e4-dockercfg-xmlrs\""
Nov 01 09:12:15.074270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.074235    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:15.113102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.113066    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h489\" (UniqueName: \"kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489\") pod \"payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:15.223893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.223812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:15.263444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.263405    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:15.638428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:15.638397    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"]
Nov 01 09:12:15.639166 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:12:15.639140    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202cc307_c196_4bfb_8a36_f56c7c4f4967.slice/crio-eec31e79cf7d5eafc09514e5a5dfafa85ef0600f923df49aa0184a15621cc1d7 WatchSource:0}: Error finding container eec31e79cf7d5eafc09514e5a5dfafa85ef0600f923df49aa0184a15621cc1d7: Status 404 returned error can't find the container with id eec31e79cf7d5eafc09514e5a5dfafa85ef0600f923df49aa0184a15621cc1d7
Nov 01 09:12:16.287106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:16.287069    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" event={"ID":"202cc307-c196-4bfb-8a36-f56c7c4f4967","Type":"ContainerStarted","Data":"62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189"}
Nov 01 09:12:16.287525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:16.287135    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:16.287525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:16.287153    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" event={"ID":"202cc307-c196-4bfb-8a36-f56c7c4f4967","Type":"ContainerStarted","Data":"eec31e79cf7d5eafc09514e5a5dfafa85ef0600f923df49aa0184a15621cc1d7"}
Nov 01 09:12:16.396918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:16.396852    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" podStartSLOduration=2.39683275 podStartE2EDuration="2.39683275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:16.396087486 +0000 UTC m=+2409.590482054" watchObservedRunningTime="2026-11-01 09:12:16.39683275 +0000 UTC m=+2409.591227306"
Nov 01 09:12:17.830779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:17.830739    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:21.376362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:21.376329    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:24.277599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:24.277562    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:27.297322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:27.297288    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:12:29.348451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:29.348411    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:33.770889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:33.770852    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:34.337903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.337865    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:34.392239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.392201    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"]
Nov 01 09:12:34.395901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.395881    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.414783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.414750    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44s5\" (UniqueName: \"kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.414937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.414824    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.414994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.414977    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.446434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.446229    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:12:34.450939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.450911    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"]
Nov 01 09:12:34.492487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.492456    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:12:34.492487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.492461    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:12:34.492713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.492509    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:12:34.516621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.516585    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44s5\" (UniqueName: \"kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.516803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.516664    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.516803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.516791    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.517499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.517474    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.519157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.519136    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.573845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.573810    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44s5\" (UniqueName: \"kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5\") pod \"maas-api-revoke-sub-64426461-cq5gm\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:34.709509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:34.709418    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:35.119701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:35.115862    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"]
Nov 01 09:12:35.406901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:35.406862    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm" event={"ID":"1c518986-7101-443a-9334-c7a10b36ffb3","Type":"ContainerStarted","Data":"badd8d620bc7c83a7129531cc69d08e64a1545ab1921b5e89f9ba055598e644e"}
Nov 01 09:12:35.407088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:35.406923    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm" event={"ID":"1c518986-7101-443a-9334-c7a10b36ffb3","Type":"ContainerStarted","Data":"edd3d3cde15ae215a34148212e074fabae6f98a3a3426d4b5aeeaabdcd8705e4"}
Nov 01 09:12:35.517107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:35.517055    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm" podStartSLOduration=1.517041278 podStartE2EDuration="1.517041278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:35.514392158 +0000 UTC m=+2428.708786711" watchObservedRunningTime="2026-11-01 09:12:35.517041278 +0000 UTC m=+2428.711435833"
Nov 01 09:12:35.918114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:35.918084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:36.129673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:36.129641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:36.271731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:36.271691    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:36.414947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:36.414916    4314 generic.go:356] "Generic (PLEG): container finished" podID="1c518986-7101-443a-9334-c7a10b36ffb3" containerID="badd8d620bc7c83a7129531cc69d08e64a1545ab1921b5e89f9ba055598e644e" exitCode=0
Nov 01 09:12:36.415133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:36.414956    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm" event={"ID":"1c518986-7101-443a-9334-c7a10b36ffb3","Type":"ContainerDied","Data":"badd8d620bc7c83a7129531cc69d08e64a1545ab1921b5e89f9ba055598e644e"}
Nov 01 09:12:37.581633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.581603    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:37.653259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.653229    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca\") pod \"1c518986-7101-443a-9334-c7a10b36ffb3\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") "
Nov 01 09:12:37.653465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.653306    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5\" (UniqueName: \"kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5\") pod \"1c518986-7101-443a-9334-c7a10b36ffb3\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") "
Nov 01 09:12:37.653465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.653350    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca\") pod \"1c518986-7101-443a-9334-c7a10b36ffb3\" (UID: \"1c518986-7101-443a-9334-c7a10b36ffb3\") "
Nov 01 09:12:37.653710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.653682    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca" pod "1c518986-7101-443a-9334-c7a10b36ffb3" (UID: "1c518986-7101-443a-9334-c7a10b36ffb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:12:37.655430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.655402    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca" pod "1c518986-7101-443a-9334-c7a10b36ffb3" (UID: "1c518986-7101-443a-9334-c7a10b36ffb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:12:37.655559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.655471    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5" pod "1c518986-7101-443a-9334-c7a10b36ffb3" (UID: "1c518986-7101-443a-9334-c7a10b36ffb3"). InnerVolumeSpecName "kube-api-access-b44s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:12:37.755153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.755114    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:37.755153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.755148    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b44s5\" (UniqueName: \"kubernetes.io/projected/1c518986-7101-443a-9334-c7a10b36ffb3-kube-api-access-b44s5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:37.755153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:37.755161    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c518986-7101-443a-9334-c7a10b36ffb3-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:38.430492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:38.430456    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"
Nov 01 09:12:38.430492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:38.430486    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm" event={"ID":"1c518986-7101-443a-9334-c7a10b36ffb3","Type":"ContainerDied","Data":"edd3d3cde15ae215a34148212e074fabae6f98a3a3426d4b5aeeaabdcd8705e4"}
Nov 01 09:12:38.430767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:38.430559    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edd3d3cde15ae215a34148212e074fabae6f98a3a3426d4b5aeeaabdcd8705e4"
Nov 01 09:12:40.460576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:40.460513    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:41.323276 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:12:41.323231    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:12:41.419851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:41.419790    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:46.341447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:46.341401    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:46.769639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:46.769609    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:52.021737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:52.021701    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:12:57.172105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:12:57.172065    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:00.954143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:00.954102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:02.310582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:02.310508    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:03.064919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:03.064887    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:07.216978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:07.216944    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:07.379784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:07.379744    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:07.523709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:07.523676    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:08.929234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:08.929200    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"]
Nov 01 09:13:09.026765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:09.026732    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"]
Nov 01 09:13:09.275441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:09.275397    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:09.471210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:09.471175    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:09.615577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:09.615421    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:11.322976 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:13:11.322934    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:13:12.687303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:12.687272    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:13.930046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:13.930005    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp" podUID="806da6f1-d584-4505-b6c4-73a2a2d60fa5" containerName="payload-processing" containerID="cri-o://67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d" gracePeriod=25
Nov 01 09:13:13.960794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:13.960760    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:14.028035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.027965    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn" podUID="90578f8b-1e7c-45fe-a5a9-a722fb8a7436" containerName="payload-pre-processing" containerID="cri-o://7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256" gracePeriod=25
Nov 01 09:13:14.199614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.199547    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:13:14.295280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.295256    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:13:14.330780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.330748    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx\" (UniqueName: \"kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx\") pod \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") "
Nov 01 09:13:14.330974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.330922    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume\") pod \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\" (UID: \"806da6f1-d584-4505-b6c4-73a2a2d60fa5\") "
Nov 01 09:13:14.331306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.331278    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume" pod "806da6f1-d584-4505-b6c4-73a2a2d60fa5" (UID: "806da6f1-d584-4505-b6c4-73a2a2d60fa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:13:14.331442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.331425    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806da6f1-d584-4505-b6c4-73a2a2d60fa5-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:14.332952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.332927    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx" pod "806da6f1-d584-4505-b6c4-73a2a2d60fa5" (UID: "806da6f1-d584-4505-b6c4-73a2a2d60fa5"). InnerVolumeSpecName "kube-api-access-7rtjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:13:14.432770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.432729    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8\" (UniqueName: \"kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8\") pod \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") "
Nov 01 09:13:14.432968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.432878    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume\") pod \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\" (UID: \"90578f8b-1e7c-45fe-a5a9-a722fb8a7436\") "
Nov 01 09:13:14.433306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.433278    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume" pod "90578f8b-1e7c-45fe-a5a9-a722fb8a7436" (UID: "90578f8b-1e7c-45fe-a5a9-a722fb8a7436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:13:14.433410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.433394    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtjx\" (UniqueName: \"kubernetes.io/projected/806da6f1-d584-4505-b6c4-73a2a2d60fa5-kube-api-access-7rtjx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:14.433475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.433412    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:14.434858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.434828    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8" pod "90578f8b-1e7c-45fe-a5a9-a722fb8a7436" (UID: "90578f8b-1e7c-45fe-a5a9-a722fb8a7436"). InnerVolumeSpecName "kube-api-access-cpxw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:13:14.534671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.534639    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxw8\" (UniqueName: \"kubernetes.io/projected/90578f8b-1e7c-45fe-a5a9-a722fb8a7436-kube-api-access-cpxw8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:14.685850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.685816    4314 generic.go:356] "Generic (PLEG): container finished" podID="806da6f1-d584-4505-b6c4-73a2a2d60fa5" containerID="67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d" exitCode=0
Nov 01 09:13:14.686020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.685857    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"
Nov 01 09:13:14.686020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.685900    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp" event={"ID":"806da6f1-d584-4505-b6c4-73a2a2d60fa5","Type":"ContainerDied","Data":"67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d"}
Nov 01 09:13:14.686020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.685946    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp" event={"ID":"806da6f1-d584-4505-b6c4-73a2a2d60fa5","Type":"ContainerDied","Data":"a5905a79ef945f7ccb7bbc631f0420dbbfec87eedb8da0971e1110e3cb18fe1e"}
Nov 01 09:13:14.686020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.685966    4314 scope.go:122] "RemoveContainer" containerID="67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d"
Nov 01 09:13:14.687371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.687347    4314 generic.go:356] "Generic (PLEG): container finished" podID="90578f8b-1e7c-45fe-a5a9-a722fb8a7436" containerID="7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256" exitCode=0
Nov 01 09:13:14.687506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.687387    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"
Nov 01 09:13:14.687506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.687418    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn" event={"ID":"90578f8b-1e7c-45fe-a5a9-a722fb8a7436","Type":"ContainerDied","Data":"7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256"}
Nov 01 09:13:14.687506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.687460    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn" event={"ID":"90578f8b-1e7c-45fe-a5a9-a722fb8a7436","Type":"ContainerDied","Data":"67ae29e3e1fa91a7c52f17232e1bd37e24a60e31d6fc8a8fc20d6a7a7958a7e6"}
Nov 01 09:13:14.696409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.696389    4314 scope.go:122] "RemoveContainer" containerID="67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d"
Nov 01 09:13:14.696702 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:13:14.696683    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d\": container with ID starting with 67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d not found: ID does not exist" containerID="67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d"
Nov 01 09:13:14.696802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.696713    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d"} err="failed to get container status \"67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d\": rpc error: code = NotFound desc = could not find container \"67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d\": container with ID starting with 67cf3e9ce0c0522280faf0fab4bc3c1653c86d72fa2d7c3a74dc7b9fdeb0d82d not found: ID does not exist"
Nov 01 09:13:14.696802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.696738    4314 scope.go:122] "RemoveContainer" containerID="7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256"
Nov 01 09:13:14.705814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.705795    4314 scope.go:122] "RemoveContainer" containerID="7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256"
Nov 01 09:13:14.706069 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:13:14.706048    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256\": container with ID starting with 7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256 not found: ID does not exist" containerID="7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256"
Nov 01 09:13:14.706128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.706080    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256"} err="failed to get container status \"7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256\": rpc error: code = NotFound desc = could not find container \"7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256\": container with ID starting with 7acc4a52fe263bdbe8cb0b12817637174765228356f0ee49449ac72a623cb256 not found: ID does not exist"
Nov 01 09:13:14.840931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.840823    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"]
Nov 01 09:13:14.854635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.854599    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-616efa-75bf59cc5c-l2ntp"]
Nov 01 09:13:14.928686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.928653    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806da6f1-d584-4505-b6c4-73a2a2d60fa5" path="/var/lib/kubelet/pods/806da6f1-d584-4505-b6c4-73a2a2d60fa5/volumes"
Nov 01 09:13:14.955467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.955434    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"]
Nov 01 09:13:14.959104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:14.959076    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-616efa-5f7f5c7847-v2tvn"]
Nov 01 09:13:15.463071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:15.463037    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:16.928970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:16.928934    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90578f8b-1e7c-45fe-a5a9-a722fb8a7436" path="/var/lib/kubelet/pods/90578f8b-1e7c-45fe-a5a9-a722fb8a7436/volumes"
Nov 01 09:13:17.872326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:17.872294    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:22.762060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:22.761979    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:22.922319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:22.922265    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:23.063213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.063099    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:23.199474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.199436    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"]
Nov 01 09:13:23.199989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.199975    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1c518986-7101-443a-9334-c7a10b36ffb3" containerName="revoke-keys"
Nov 01 09:13:23.200041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.199990    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c518986-7101-443a-9334-c7a10b36ffb3" containerName="revoke-keys"
Nov 01 09:13:23.200041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200010    4314 cpu_manager.go:414] "containerMap: removing container" podUID="90578f8b-1e7c-45fe-a5a9-a722fb8a7436" containerName="payload-pre-processing"
Nov 01 09:13:23.200041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200015    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90578f8b-1e7c-45fe-a5a9-a722fb8a7436" containerName="payload-pre-processing"
Nov 01 09:13:23.200041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200022    4314 cpu_manager.go:414] "containerMap: removing container" podUID="806da6f1-d584-4505-b6c4-73a2a2d60fa5" containerName="payload-processing"
Nov 01 09:13:23.200041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200027    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806da6f1-d584-4505-b6c4-73a2a2d60fa5" containerName="payload-processing"
Nov 01 09:13:23.200220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200111    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c518986-7101-443a-9334-c7a10b36ffb3" containerName="revoke-keys"
Nov 01 09:13:23.200220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200123    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="90578f8b-1e7c-45fe-a5a9-a722fb8a7436" containerName="payload-pre-processing"
Nov 01 09:13:23.200220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.200131    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="806da6f1-d584-4505-b6c4-73a2a2d60fa5" containerName="payload-processing"
Nov 01 09:13:23.205051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.205030    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.235214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.235185    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgp6\" (UniqueName: \"kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.235369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.235252    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.235424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.235362    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.254219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.253892    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:13:23.254405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.254247    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:13:23.258386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.258362    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"]
Nov 01 09:13:23.299837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.299799    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:13:23.299837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.299831    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:13:23.336799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.336705    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.336973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.336796    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.336973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.336965    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgp6\" (UniqueName: \"kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.337469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.337439    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.339090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.339072    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.394595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.394561    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgp6\" (UniqueName: \"kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6\") pod \"maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.518234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.518199    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:23.884308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:23.884270    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"]
Nov 01 09:13:23.885752 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:13:23.885724    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87627620_7882_4622_8f09_8aeae4d9c05f.slice/crio-be238041af7c1354b4d29ad490c62ec1f5e8036ae4ff1f52416f96dfca7d5eec WatchSource:0}: Error finding container be238041af7c1354b4d29ad490c62ec1f5e8036ae4ff1f52416f96dfca7d5eec: Status 404 returned error can't find the container with id be238041af7c1354b4d29ad490c62ec1f5e8036ae4ff1f52416f96dfca7d5eec
Nov 01 09:13:24.758396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:24.758358    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph" event={"ID":"87627620-7882-4622-8f09-8aeae4d9c05f","Type":"ContainerStarted","Data":"9a97aea7bc8aa5ff361c28da2be51be0f21afd15c6a9828a710ae96852f5245f"}
Nov 01 09:13:24.758522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:24.758429    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph" event={"ID":"87627620-7882-4622-8f09-8aeae4d9c05f","Type":"ContainerStarted","Data":"be238041af7c1354b4d29ad490c62ec1f5e8036ae4ff1f52416f96dfca7d5eec"}
Nov 01 09:13:24.868362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:24.868305    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph" podStartSLOduration=1.8682881770000002 podStartE2EDuration="1.868288177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:13:24.8656558 +0000 UTC m=+2478.060050356" watchObservedRunningTime="2026-11-01 09:13:24.868288177 +0000 UTC m=+2478.062682722"
Nov 01 09:13:25.781556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:25.780976    4314 generic.go:356] "Generic (PLEG): container finished" podID="87627620-7882-4622-8f09-8aeae4d9c05f" containerID="9a97aea7bc8aa5ff361c28da2be51be0f21afd15c6a9828a710ae96852f5245f" exitCode=0
Nov 01 09:13:25.781556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:25.781049    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph" event={"ID":"87627620-7882-4622-8f09-8aeae4d9c05f","Type":"ContainerDied","Data":"9a97aea7bc8aa5ff361c28da2be51be0f21afd15c6a9828a710ae96852f5245f"}
Nov 01 09:13:26.937817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.937791    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:26.989775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.989746    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca\") pod \"87627620-7882-4622-8f09-8aeae4d9c05f\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") "
Nov 01 09:13:26.989936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.989881    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca\") pod \"87627620-7882-4622-8f09-8aeae4d9c05f\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") "
Nov 01 09:13:26.990001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.989947    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6\" (UniqueName: \"kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6\") pod \"87627620-7882-4622-8f09-8aeae4d9c05f\" (UID: \"87627620-7882-4622-8f09-8aeae4d9c05f\") "
Nov 01 09:13:26.990093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.990069    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca" pod "87627620-7882-4622-8f09-8aeae4d9c05f" (UID: "87627620-7882-4622-8f09-8aeae4d9c05f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:13:26.990351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.990335    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:26.992034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.992015    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca" pod "87627620-7882-4622-8f09-8aeae4d9c05f" (UID: "87627620-7882-4622-8f09-8aeae4d9c05f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:13:26.992158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:26.992089    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6" pod "87627620-7882-4622-8f09-8aeae4d9c05f" (UID: "87627620-7882-4622-8f09-8aeae4d9c05f"). InnerVolumeSpecName "kube-api-access-ktgp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:13:27.091818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:27.091741    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87627620-7882-4622-8f09-8aeae4d9c05f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:27.091818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:27.091768    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktgp6\" (UniqueName: \"kubernetes.io/projected/87627620-7882-4622-8f09-8aeae4d9c05f-kube-api-access-ktgp6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:27.793961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:27.793929    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"
Nov 01 09:13:27.793961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:27.793939    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph" event={"ID":"87627620-7882-4622-8f09-8aeae4d9c05f","Type":"ContainerDied","Data":"be238041af7c1354b4d29ad490c62ec1f5e8036ae4ff1f52416f96dfca7d5eec"}
Nov 01 09:13:27.794184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:27.793998    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be238041af7c1354b4d29ad490c62ec1f5e8036ae4ff1f52416f96dfca7d5eec"
Nov 01 09:13:27.893280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:27.893246    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:32.371894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:32.371859    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"]
Nov 01 09:13:32.372416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:32.372071    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" containerID="cri-o://1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405" gracePeriod=30
Nov 01 09:13:32.693467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:32.693384    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:32.892698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:32.892660    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:35.674496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:35.674457    4314 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:13:35.674928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:35.674550    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:13:37.641323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.641261    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 09:13:37.694606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.694577    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:37.823775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.823743    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.823940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.823820    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.823989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.823951    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824000    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824034    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824067    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824082    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:13:37.824160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824102    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824155    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824229    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket\") pod \"9358e53e-caf4-42da-be9c-2d18e49e5416\" (UID: \"9358e53e-caf4-42da-be9c-2d18e49e5416\") "
Nov 01 09:13:37.824555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824512    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:13:37.824837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824757    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:13:37.824837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824792    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.824837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824817    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.825163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.824946    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:13:37.825240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.825164    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:13:37.826640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.826615    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:13:37.826872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.826849    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:13:37.826976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.826966    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "kube-api-access-5qnbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:13:37.827070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.827054    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo" pod "9358e53e-caf4-42da-be9c-2d18e49e5416" (UID: "9358e53e-caf4-42da-be9c-2d18e49e5416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:13:37.855722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.855693    4314 generic.go:356] "Generic (PLEG): container finished" podID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerID="1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405" exitCode=0
Nov 01 09:13:37.855836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.855744    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"
Nov 01 09:13:37.855836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.855760    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" event={"ID":"9358e53e-caf4-42da-be9c-2d18e49e5416","Type":"ContainerDied","Data":"1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405"}
Nov 01 09:13:37.855836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.855804    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h" event={"ID":"9358e53e-caf4-42da-be9c-2d18e49e5416","Type":"ContainerDied","Data":"0e9251a437e3afab7af037efdd3c6a9df2e222891d6082f728cd20eed7c2add9"}
Nov 01 09:13:37.855836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.855827    4314 scope.go:122] "RemoveContainer" containerID="1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405"
Nov 01 09:13:37.865727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.865709    4314 scope.go:122] "RemoveContainer" containerID="1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405"
Nov 01 09:13:37.865980 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:13:37.865965    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405\": container with ID starting with 1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405 not found: ID does not exist" containerID="1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405"
Nov 01 09:13:37.866026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.865988    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405"} err="failed to get container status \"1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405\": rpc error: code = NotFound desc = could not find container \"1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405\": container with ID starting with 1a020006a117c9b1fbc4ecaf0bd913fd3f68d06aa8251513d0603f20a689f405 not found: ID does not exist"
Nov 01 09:13:37.925653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925588    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnbv\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-kube-api-access-5qnbv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.925653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925614    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.925653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925623    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.925653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925634    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358e53e-caf4-42da-be9c-2d18e49e5416-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.925653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925643    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358e53e-caf4-42da-be9c-2d18e49e5416-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.925653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925651    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:37.925941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:37.925661    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358e53e-caf4-42da-be9c-2d18e49e5416-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:13:38.010948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:38.010910    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"]
Nov 01 09:13:38.016851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:38.016825    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-dnm6h"]
Nov 01 09:13:38.928395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:38.928363    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" path="/var/lib/kubelet/pods/9358e53e-caf4-42da-be9c-2d18e49e5416/volumes"
Nov 01 09:13:41.322864 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:13:41.322830    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:13:44.146839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:44.146803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:48.327524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:48.327493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:50.411793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:50.411751    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:51.133942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:51.133904    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:53.190119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:53.190084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:55.383163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:55.383124    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:56.925523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:56.925486    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:57.073322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:57.073283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:58.503034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:58.502993    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:59.085032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:59.084094    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:59.113627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:59.113596    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"]
Nov 01 09:13:59.113861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:59.113835    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy" containerID="cri-o://46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc" gracePeriod=30
Nov 01 09:13:59.224916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:59.224880    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:13:59.883272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:13:59.883235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:00.364176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:00.364146    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:00.604878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:00.604797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:00.747523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:00.747475    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:01.027477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:01.027441    4314 generic.go:356] "Generic (PLEG): container finished" podID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerID="3bd022a8e6d6be4c87b232f62957af32e31ce44fc4e57509feae71ae0c7e49fe" exitCode=0
Nov 01 09:14:01.027973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:01.027513    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" event={"ID":"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f","Type":"ContainerDied","Data":"3bd022a8e6d6be4c87b232f62957af32e31ce44fc4e57509feae71ae0c7e49fe"}
Nov 01 09:14:01.027973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:01.027955    4314 scope.go:122] "RemoveContainer" containerID="3bd022a8e6d6be4c87b232f62957af32e31ce44fc4e57509feae71ae0c7e49fe"
Nov 01 09:14:01.029484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:01.029448    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:01.029586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:01.029559    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:01.029635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:01.029604    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:02.037090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:02.037051    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" event={"ID":"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f","Type":"ContainerStarted","Data":"90abec001d136c82a16c65ff6a9cf709d5123fdbafdcf6f76802d60a9497249e"}
Nov 01 09:14:02.441957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:02.441919    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:14:02.442178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:02.442148    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 09:14:02.442247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:02.442224    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 09:14:03.442396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:03.442359    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 09:14:03.442842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:03.442407    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 09:14:04.402126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.401868    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:14:04.442156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.442120    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 09:14:04.442337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.442189    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 09:14:04.496450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496416    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496469    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496513    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496557    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496578    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496606    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496702    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496725    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496753    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo\") pod \"6d67fde0-b276-4274-a498-f18d3308336b\" (UID: \"6d67fde0-b276-4274-a498-f18d3308336b\") "
Nov 01 09:14:04.496976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.496802    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:04.497427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.497341    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.497730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.497527    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:04.497730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.497576    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:04.497730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.497693    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:04.497947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.497815    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:04.499467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.499441    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:04.499692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.499668    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:14:04.500580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.500559    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "kube-api-access-4dznx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:04.501205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.501160    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy" pod "6d67fde0-b276-4274-a498-f18d3308336b" (UID: "6d67fde0-b276-4274-a498-f18d3308336b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598688    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dznx\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-kube-api-access-4dznx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598724    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598739    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598754    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d67fde0-b276-4274-a498-f18d3308336b-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598768    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598786    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d67fde0-b276-4274-a498-f18d3308336b-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598800    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d67fde0-b276-4274-a498-f18d3308336b-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:04.599012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:04.598814    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d67fde0-b276-4274-a498-f18d3308336b-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:05.023716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.023650    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:05.070426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.068634    4314 generic.go:356] "Generic (PLEG): container finished" podID="6d67fde0-b276-4274-a498-f18d3308336b" containerID="46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc" exitCode=0
Nov 01 09:14:05.070426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.068760    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" event={"ID":"6d67fde0-b276-4274-a498-f18d3308336b","Type":"ContainerDied","Data":"46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc"}
Nov 01 09:14:05.070426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.068816    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g" event={"ID":"6d67fde0-b276-4274-a498-f18d3308336b","Type":"ContainerDied","Data":"320a1ac87143c8a07b42b48bf09455ad43c7d2d1d3e15aca89f067c8a8d85eb7"}
Nov 01 09:14:05.070426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.068848    4314 scope.go:122] "RemoveContainer" containerID="46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc"
Nov 01 09:14:05.070426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.069034    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"
Nov 01 09:14:05.090553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.090514    4314 scope.go:122] "RemoveContainer" containerID="653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"
Nov 01 09:14:05.100423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.100396    4314 scope.go:122] "RemoveContainer" containerID="46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc"
Nov 01 09:14:05.100776 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:05.100754    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc\": container with ID starting with 46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc not found: ID does not exist" containerID="46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc"
Nov 01 09:14:05.100880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.100784    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc"} err="failed to get container status \"46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc\": rpc error: code = NotFound desc = could not find container \"46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc\": container with ID starting with 46171754bc0baddc46304268ab7b3c1befacac4b838ed09de3c223a1b02832bc not found: ID does not exist"
Nov 01 09:14:05.100880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.100804    4314 scope.go:122] "RemoveContainer" containerID="653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"
Nov 01 09:14:05.101116 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:05.101093    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548\": container with ID starting with 653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548 not found: ID does not exist" containerID="653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"
Nov 01 09:14:05.101188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.101125    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548"} err="failed to get container status \"653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548\": rpc error: code = NotFound desc = could not find container \"653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548\": container with ID starting with 653fd624b76ba2c121380f067ee9e74ccd19661cc42266a54be00ec939205548 not found: ID does not exist"
Nov 01 09:14:05.286319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.286286    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"]
Nov 01 09:14:05.289026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.289004    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-616efa-openshift-default-d64fb7989-xzp5g"]
Nov 01 09:14:05.441879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.441838    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 09:14:05.442071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.441899    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 09:14:05.685392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.685288    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:05.981634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:05.980450    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:06.286133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.286025    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"]
Nov 01 09:14:06.383854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.383814    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"]
Nov 01 09:14:06.441621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.441583    4314 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 09:14:06.441802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.441637    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 09:14:06.622734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.622632    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:06.786169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.786124    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:06.929008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:06.928928    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d67fde0-b276-4274-a498-f18d3308336b" path="/var/lib/kubelet/pods/6d67fde0-b276-4274-a498-f18d3308336b/volumes"
Nov 01 09:14:07.446498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:07.446471    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:14:07.446947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:07.446917    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:14:07.447566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:07.447548    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:14:10.811570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:10.811521    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:11.287018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.286958    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf" podUID="bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" containerName="payload-processing" containerID="cri-o://91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7" gracePeriod=25
Nov 01 09:14:11.323645 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:11.323603    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:14:11.384356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.384309    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp" podUID="3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" containerName="payload-pre-processing" containerID="cri-o://e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a" gracePeriod=25
Nov 01 09:14:11.556618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.556555    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:14:11.587982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.587952    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6\" (UniqueName: \"kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6\") pod \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") "
Nov 01 09:14:11.588171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.588029    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume\") pod \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\" (UID: \"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f\") "
Nov 01 09:14:11.588743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.588704    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume" pod "bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" (UID: "bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:11.590827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.590802    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6" pod "bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" (UID: "bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f"). InnerVolumeSpecName "kube-api-access-5nrr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:11.641144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.641115    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:14:11.674294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.674262    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:11.688870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.688831    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r\" (UniqueName: \"kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r\") pod \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") "
Nov 01 09:14:11.689071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.689046    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume\") pod \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\" (UID: \"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b\") "
Nov 01 09:14:11.689410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.689379    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume" pod "3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" (UID: "3edc26dc-fd9b-46cf-b31d-eb3ec788a84b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:11.689721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.689696    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:11.689821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.689726    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrr6\" (UniqueName: \"kubernetes.io/projected/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-kube-api-access-5nrr6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:11.689821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.689742    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:11.691112 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.691086    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r" pod "3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" (UID: "3edc26dc-fd9b-46cf-b31d-eb3ec788a84b"). InnerVolumeSpecName "kube-api-access-v8t8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:11.790983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:11.790947    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8t8r\" (UniqueName: \"kubernetes.io/projected/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b-kube-api-access-v8t8r\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:12.134821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.134787    4314 generic.go:356] "Generic (PLEG): container finished" podID="3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" containerID="e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a" exitCode=0
Nov 01 09:14:12.135351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.134838    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"
Nov 01 09:14:12.135351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.134867    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp" event={"ID":"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b","Type":"ContainerDied","Data":"e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a"}
Nov 01 09:14:12.135351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.134923    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp" event={"ID":"3edc26dc-fd9b-46cf-b31d-eb3ec788a84b","Type":"ContainerDied","Data":"c36f3b902eac431740be5cf14f6ac6a2fe6317911f75226b2807c16f41c663b7"}
Nov 01 09:14:12.135351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.134949    4314 scope.go:122] "RemoveContainer" containerID="e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a"
Nov 01 09:14:12.136488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.136460    4314 generic.go:356] "Generic (PLEG): container finished" podID="bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" containerID="91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7" exitCode=0
Nov 01 09:14:12.136656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.136506    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"
Nov 01 09:14:12.136656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.136571    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf" event={"ID":"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f","Type":"ContainerDied","Data":"91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7"}
Nov 01 09:14:12.136656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.136616    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf" event={"ID":"bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f","Type":"ContainerDied","Data":"ec683a4f84e5e9b91727081e728a9ed9407e608723d3aa375f9e17b7a7b4957b"}
Nov 01 09:14:12.144962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.144938    4314 scope.go:122] "RemoveContainer" containerID="e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a"
Nov 01 09:14:12.145268 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:12.145247    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a\": container with ID starting with e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a not found: ID does not exist" containerID="e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a"
Nov 01 09:14:12.145338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.145276    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a"} err="failed to get container status \"e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a\": rpc error: code = NotFound desc = could not find container \"e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a\": container with ID starting with e3ce4e1c62a1551a71e3525194d1e7a33ea5004f51143813d25552a721a0b56a not found: ID does not exist"
Nov 01 09:14:12.145338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.145294    4314 scope.go:122] "RemoveContainer" containerID="91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7"
Nov 01 09:14:12.154407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.154386    4314 scope.go:122] "RemoveContainer" containerID="91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7"
Nov 01 09:14:12.154661 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:12.154642    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7\": container with ID starting with 91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7 not found: ID does not exist" containerID="91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7"
Nov 01 09:14:12.154724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.154667    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7"} err="failed to get container status \"91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7\": rpc error: code = NotFound desc = could not find container \"91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7\": container with ID starting with 91094f006b2862c5b9abd6c0069e8ac291dab49417ce41a3b8173ea24400d0d7 not found: ID does not exist"
Nov 01 09:14:12.288555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.288508    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"]
Nov 01 09:14:12.292994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.292964    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9f4e9b-76c6555b7-mk7hf"]
Nov 01 09:14:12.394433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.394352    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"]
Nov 01 09:14:12.397959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.397934    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9f4e9b-85f8dbd7f-xlrjp"]
Nov 01 09:14:12.928389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.928355    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" path="/var/lib/kubelet/pods/3edc26dc-fd9b-46cf-b31d-eb3ec788a84b/volumes"
Nov 01 09:14:12.928792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:12.928775    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" path="/var/lib/kubelet/pods/bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f/volumes"
Nov 01 09:14:13.887099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:13.887013    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:14.031428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:14.031391    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:14.303792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:14.303755    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:15.922284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:15.922243    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:17.752048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:17.752011    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"]
Nov 01 09:14:17.752554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:17.752304    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" containerID="cri-o://a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627" gracePeriod=30
Nov 01 09:14:20.929176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:20.929148    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"]
Nov 01 09:14:20.929613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:20.929343    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" podUID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" containerName="discovery" containerID="cri-o://7b82e52b807787d0fb05751d51393da59310513e5b7a83b3cb8e2cd983e87171" gracePeriod=30
Nov 01 09:14:21.107649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.107473    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:21.194451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.194359    4314 generic.go:356] "Generic (PLEG): container finished" podID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" containerID="7b82e52b807787d0fb05751d51393da59310513e5b7a83b3cb8e2cd983e87171" exitCode=0
Nov 01 09:14:21.194451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.194431    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" event={"ID":"a8fa35a9-077b-43d2-bd27-ee185f40f9cd","Type":"ContainerDied","Data":"7b82e52b807787d0fb05751d51393da59310513e5b7a83b3cb8e2cd983e87171"}
Nov 01 09:14:21.232935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.232909    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 09:14:21.298561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298509    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.298767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298630    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.298767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298663    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.298767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298718    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.298767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298740    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.299002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298836    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.299002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.298887    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token\") pod \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\" (UID: \"a8fa35a9-077b-43d2-bd27-ee185f40f9cd\") "
Nov 01 09:14:21.299272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.299236    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:21.299612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.299563    4314 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-ca-configmap\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.301028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.301005    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:21.301154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.301055    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:14:21.301303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.301282    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "kube-api-access-zjzft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:21.301366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.301348    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:14:21.301508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.301486    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:14:21.301508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.301499    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token" pod "a8fa35a9-077b-43d2-bd27-ee185f40f9cd" (UID: "a8fa35a9-077b-43d2-bd27-ee185f40f9cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:21.334246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.334218    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:21.401176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.401137    4314 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-local-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.401176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.401171    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzft\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-kube-api-access-zjzft\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.401394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.401191    4314 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-csr-dns-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.401394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.401203    4314 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-cacerts\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.401394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.401215    4314 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-kubeconfig\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.401394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.401228    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fa35a9-077b-43d2-bd27-ee185f40f9cd-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:21.699857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.699821    4314 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:14:21.700028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:21.699885    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:14:22.201726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.201694    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"
Nov 01 09:14:22.202166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.201694    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst" event={"ID":"a8fa35a9-077b-43d2-bd27-ee185f40f9cd","Type":"ContainerDied","Data":"88effa0d450fbfe711799dd60a48faa6a9f37cf4df2b9d6653f2981191d48d39"}
Nov 01 09:14:22.202166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.201830    4314 scope.go:122] "RemoveContainer" containerID="7b82e52b807787d0fb05751d51393da59310513e5b7a83b3cb8e2cd983e87171"
Nov 01 09:14:22.402243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.402205    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"]
Nov 01 09:14:22.410451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.410423    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5vst"]
Nov 01 09:14:22.469150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.469055    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:22.931224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.931191    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" path="/var/lib/kubelet/pods/a8fa35a9-077b-43d2-bd27-ee185f40f9cd/volumes"
Nov 01 09:14:22.952867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.952836    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"]
Nov 01 09:14:22.953355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953338    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" containerName="discovery"
Nov 01 09:14:22.953355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953355    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" containerName="discovery"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953369    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953375    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953398    4314 cpu_manager.go:414] "containerMap: removing container" podUID="3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" containerName="payload-pre-processing"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953407    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" containerName="payload-pre-processing"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953426    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" containerName="payload-processing"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953432    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" containerName="payload-processing"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953439    4314 cpu_manager.go:414] "containerMap: removing container" podUID="87627620-7882-4622-8f09-8aeae4d9c05f" containerName="revoke-keys"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953444    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87627620-7882-4622-8f09-8aeae4d9c05f" containerName="revoke-keys"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953454    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy"
Nov 01 09:14:22.953518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953458    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953543    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd06ef19-c1f9-49e0-9eda-0d4fccb1da2f" containerName="payload-processing"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953556    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953565    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="87627620-7882-4622-8f09-8aeae4d9c05f" containerName="revoke-keys"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953572    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9358e53e-caf4-42da-be9c-2d18e49e5416" containerName="istio-proxy"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953579    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fa35a9-077b-43d2-bd27-ee185f40f9cd" containerName="discovery"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953584    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy"
Nov 01 09:14:22.954032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.953594    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edc26dc-fd9b-46cf-b31d-eb3ec788a84b" containerName="payload-pre-processing"
Nov 01 09:14:22.971051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:22.971018    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.014799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.014757    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"]
Nov 01 09:14:23.018583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.018557    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 09:14:23.019872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.019840    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.019927    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.019988    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.020060    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.020097    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kzrl\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.020176    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.020217    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.020578    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.020850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.020807    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.036664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.036638    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:23.056083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.056065    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c1c173-openshift-default-dockercfg-tpqcx\""
Nov 01 09:14:23.122589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.122494    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.123090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.123065    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.123281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.123266    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.123610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.123593    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.123856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.123826    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.124007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.123978    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.124166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.124134    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.124323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.124284    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.124565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.124547    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket\") pod \"0e08e935-62b5-4295-840b-6ba4f67c27b3\" (UID: \"0e08e935-62b5-4295-840b-6ba4f67c27b3\") "
Nov 01 09:14:23.124915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.124893    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127656    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kzrl\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127790    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127837    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127886    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127919    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.128071    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.128160    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.128236    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.128350    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.126798    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127063    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:23.128617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.127437    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:23.129685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.129129    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:23.129685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.129429    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.130435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.130405    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.130695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.130672    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.130963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.130940    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.131296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.131263    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:23.141771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.141743    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.143426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.143399    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:23.143800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.143519    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:23.143800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.143745    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "kube-api-access-qfbgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:23.145378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.145312    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo" pod "0e08e935-62b5-4295-840b-6ba4f67c27b3" (UID: "0e08e935-62b5-4295-840b-6ba4f67c27b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:14:23.145625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.145600    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.218770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.218735    4314 generic.go:356] "Generic (PLEG): container finished" podID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerID="a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627" exitCode=0
Nov 01 09:14:23.218770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.218773    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" event={"ID":"0e08e935-62b5-4295-840b-6ba4f67c27b3","Type":"ContainerDied","Data":"a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627"}
Nov 01 09:14:23.219359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.218824    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"
Nov 01 09:14:23.219359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.218849    4314 scope.go:122] "RemoveContainer" containerID="a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627"
Nov 01 09:14:23.219359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.218836    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8" event={"ID":"0e08e935-62b5-4295-840b-6ba4f67c27b3","Type":"ContainerDied","Data":"d6938f050bb445c4067aadda7c1b445a3257fb3db4997324b73cf608308df803"}
Nov 01 09:14:23.219846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.219825    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.227366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.227344    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kzrl\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl\") pod \"e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") " pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.229731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229704    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.229807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229738    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.229807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229756    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.229807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229770    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.229807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229785    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08e935-62b5-4295-840b-6ba4f67c27b3-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.229807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229800    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbgl\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-kube-api-access-qfbgl\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.230006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229816    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e08e935-62b5-4295-840b-6ba4f67c27b3-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.230006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229829    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.230006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.229843    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e08e935-62b5-4295-840b-6ba4f67c27b3-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:23.230199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.230077    4314 scope.go:122] "RemoveContainer" containerID="a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"
Nov 01 09:14:23.239417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.239398    4314 scope.go:122] "RemoveContainer" containerID="a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627"
Nov 01 09:14:23.239764 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:23.239738    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627\": container with ID starting with a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627 not found: ID does not exist" containerID="a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627"
Nov 01 09:14:23.239927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.239771    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627"} err="failed to get container status \"a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627\": rpc error: code = NotFound desc = could not find container \"a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627\": container with ID starting with a79a4450e6db7b9a1f8ad4916d7bb9daae1141996fdcb02446ae116c67330627 not found: ID does not exist"
Nov 01 09:14:23.239927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.239791    4314 scope.go:122] "RemoveContainer" containerID="a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"
Nov 01 09:14:23.240113 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:23.240095    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae\": container with ID starting with a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae not found: ID does not exist" containerID="a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"
Nov 01 09:14:23.240163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.240116    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae"} err="failed to get container status \"a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae\": rpc error: code = NotFound desc = could not find container \"a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae\": container with ID starting with a459ccc1212376a10447d86a24d1e2c0bc6c0328e37d933a818f86a349da7dae not found: ID does not exist"
Nov 01 09:14:23.285988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.285912    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:23.336694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.336646    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:23.387107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.387075    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"]
Nov 01 09:14:23.393833 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.393803    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-wbgq8"]
Nov 01 09:14:23.707621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.707597    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"]
Nov 01 09:14:23.708869 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:14:23.708843    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c92efb_3798_491a_ac10_165626a4f87f.slice/crio-95ada2f3e03d70b85643312ec3699efc0384f99e6a669cfac273b3903f4c818a WatchSource:0}: Error finding container 95ada2f3e03d70b85643312ec3699efc0384f99e6a669cfac273b3903f4c818a: Status 404 returned error can't find the container with id 95ada2f3e03d70b85643312ec3699efc0384f99e6a669cfac273b3903f4c818a
Nov 01 09:14:23.711508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.711472    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:23.711597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.711563    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:23.711646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:23.711595    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:24.227107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.227071    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" event={"ID":"16c92efb-3798-491a-ac10-165626a4f87f","Type":"ContainerStarted","Data":"99e471e65f07de1b92cdabc5e02cdc5360e9763c76bcb39dd783af68855ce64e"}
Nov 01 09:14:24.227568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.227129    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" event={"ID":"16c92efb-3798-491a-ac10-165626a4f87f","Type":"ContainerStarted","Data":"95ada2f3e03d70b85643312ec3699efc0384f99e6a669cfac273b3903f4c818a"}
Nov 01 09:14:24.286979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.286940    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:24.288422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.288394    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 09:14:24.288572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.288444    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 09:14:24.341986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.341921    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podStartSLOduration=2.341901673 podStartE2EDuration="2.341901673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:14:24.339741504 +0000 UTC m=+2537.534136059" watchObservedRunningTime="2026-11-01 09:14:24.341901673 +0000 UTC m=+2537.536296229"
Nov 01 09:14:24.928288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:24.928252    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" path="/var/lib/kubelet/pods/0e08e935-62b5-4295-840b-6ba4f67c27b3/volumes"
Nov 01 09:14:25.291114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:25.291082    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:26.239035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:26.239004    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:26.240008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:26.239987    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:14:26.440711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:26.440678    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:27.212324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:27.212281    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:27.377412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:27.377381    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:27.518861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:27.518829    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:30.352725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.352688    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:30.518212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.518172    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:30.762918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.762885    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"]
Nov 01 09:14:30.763398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763385    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy"
Nov 01 09:14:30.763450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763401    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy"
Nov 01 09:14:30.763450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763414    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy"
Nov 01 09:14:30.763450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763419    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy"
Nov 01 09:14:30.763450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763446    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy"
Nov 01 09:14:30.763599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763452    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d67fde0-b276-4274-a498-f18d3308336b" containerName="istio-proxy"
Nov 01 09:14:30.763599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763552    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy"
Nov 01 09:14:30.763599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.763564    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e08e935-62b5-4295-840b-6ba4f67c27b3" containerName="istio-proxy"
Nov 01 09:14:30.767249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.767233    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:30.820812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.820778    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"]
Nov 01 09:14:30.885558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.864913    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c1c173\""
Nov 01 09:14:30.885558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.866438    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c1c173-dockercfg-lmf45\""
Nov 01 09:14:30.919041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.918999    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fm6\" (UniqueName: \"kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6\") pod \"payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:30.919221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.919136    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:30.929757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.929728    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"]
Nov 01 09:14:30.933540 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.933508    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:30.981886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:30.981849    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"]
Nov 01 09:14:31.020959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.020865    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fm6\" (UniqueName: \"kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6\") pod \"payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:31.021144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.020957    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:31.021144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.021028    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.021144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.021069    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7sn\" (UniqueName: \"kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn\") pod \"payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.021947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.021917    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:31.078279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.078246    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fm6\" (UniqueName: \"kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6\") pod \"payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:31.081116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.081093    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:31.122178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.122098    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.122178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.122163    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7sn\" (UniqueName: \"kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn\") pod \"payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.122954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.122924    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.179592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.179554    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7sn\" (UniqueName: \"kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn\") pod \"payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.247508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.247473    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:31.448663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.448628    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"]
Nov 01 09:14:31.449969 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:14:31.449936    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d957b9_3510_48b0_b31f_372e9f55f508.slice/crio-282e39eb2ef6a3c008e3b217dc8fa72933fea02e9aa2340701ee379988fc6080 WatchSource:0}: Error finding container 282e39eb2ef6a3c008e3b217dc8fa72933fea02e9aa2340701ee379988fc6080: Status 404 returned error can't find the container with id 282e39eb2ef6a3c008e3b217dc8fa72933fea02e9aa2340701ee379988fc6080
Nov 01 09:14:31.614807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.614777    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"]
Nov 01 09:14:31.616218 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:14:31.616192    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e1e614_7c20_4e13_a58c_330f768c6867.slice/crio-773d06e3a17c7a6702f5008630a140af1872c6588fe4724d7c1b373f62668e17 WatchSource:0}: Error finding container 773d06e3a17c7a6702f5008630a140af1872c6588fe4724d7c1b373f62668e17: Status 404 returned error can't find the container with id 773d06e3a17c7a6702f5008630a140af1872c6588fe4724d7c1b373f62668e17
Nov 01 09:14:31.713917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:31.713887    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:32.275406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.275351    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4" event={"ID":"73d957b9-3510-48b0-b31f-372e9f55f508","Type":"ContainerStarted","Data":"0a9030f7131219a927cafa3f4c4bc8143a3bf230e9fd3151b1c85857e737165a"}
Nov 01 09:14:32.275630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.275456    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:32.275630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.275480    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4" event={"ID":"73d957b9-3510-48b0-b31f-372e9f55f508","Type":"ContainerStarted","Data":"282e39eb2ef6a3c008e3b217dc8fa72933fea02e9aa2340701ee379988fc6080"}
Nov 01 09:14:32.277104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.277071    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69" event={"ID":"04e1e614-7c20-4e13-a58c-330f768c6867","Type":"ContainerStarted","Data":"238cf4745bb10475527ed047bb5191ad276c323db299779a3c55357f2ec6226b"}
Nov 01 09:14:32.277214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.277131    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69" event={"ID":"04e1e614-7c20-4e13-a58c-330f768c6867","Type":"ContainerStarted","Data":"773d06e3a17c7a6702f5008630a140af1872c6588fe4724d7c1b373f62668e17"}
Nov 01 09:14:32.277214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.277152    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:32.417382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.417325    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4" podStartSLOduration=2.417310709 podStartE2EDuration="2.417310709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:14:32.416216513 +0000 UTC m=+2545.610611070" watchObservedRunningTime="2026-11-01 09:14:32.417310709 +0000 UTC m=+2545.611705264"
Nov 01 09:14:32.527049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:32.526948    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69" podStartSLOduration=2.526932007 podStartE2EDuration="2.526932007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:14:32.523699462 +0000 UTC m=+2545.718094029" watchObservedRunningTime="2026-11-01 09:14:32.526932007 +0000 UTC m=+2545.721326562"
Nov 01 09:14:34.146338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:34.146300    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:35.401155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.401121    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:35.784342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.784308    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"]
Nov 01 09:14:35.788393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.788370    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.837282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.837252    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-514eb3-openshift-default-dockercfg-d5q27\""
Nov 01 09:14:35.870693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.870650    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:35.881350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881311    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7nbg\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881450    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881520    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881658    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881712    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881743    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881907    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.881962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.881951    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.888125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.888096    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"]
Nov 01 09:14:35.983590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983553    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.983590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983595    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.983864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983624    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.983864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983653    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.983864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983691    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.983864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983729    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.983864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983841    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983881    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7nbg\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.983905    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.984006    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.984133    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.984213    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.984328    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.984486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.984466    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.986228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.986203    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:35.986328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:35.986245    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:36.020706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.020657    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"]
Nov 01 09:14:36.020915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.020894    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy" containerID="cri-o://fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4" gracePeriod=30
Nov 01 09:14:36.056272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.056199    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:36.056446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.056237    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7nbg\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg\") pod \"e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") " pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:36.092418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.092363    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:36.102433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.102405    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:36.471594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.471464    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:36.523281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.523256    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"]
Nov 01 09:14:36.524368 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:14:36.524343    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d7ab31_8cf7_4f36_b7b4_5925c6fd90e0.slice/crio-d478b634e505944b367afc863c04242ea0dd4f9fa10580f795d2ea0a4946d1d5 WatchSource:0}: Error finding container d478b634e505944b367afc863c04242ea0dd4f9fa10580f795d2ea0a4946d1d5: Status 404 returned error can't find the container with id d478b634e505944b367afc863c04242ea0dd4f9fa10580f795d2ea0a4946d1d5
Nov 01 09:14:36.526904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.526866    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:36.527004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.526946    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:36.527004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:36.526994    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:14:37.057684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:37.057626    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:37.201728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:37.201680    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:37.308861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:37.308780    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" event={"ID":"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0","Type":"ContainerStarted","Data":"55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"}
Nov 01 09:14:37.308861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:37.308851    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" event={"ID":"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0","Type":"ContainerStarted","Data":"d478b634e505944b367afc863c04242ea0dd4f9fa10580f795d2ea0a4946d1d5"}
Nov 01 09:14:37.428554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:37.425266    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podStartSLOduration=2.425250006 podStartE2EDuration="2.425250006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:14:37.424030767 +0000 UTC m=+2550.618425332" watchObservedRunningTime="2026-11-01 09:14:37.425250006 +0000 UTC m=+2550.619644582"
Nov 01 09:14:37.726577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:37.726476    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:38.103007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.102972    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:38.108528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.108501    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:38.266582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.266551    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:38.318605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.318573    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:38.319712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.319685    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:14:38.491884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.491849    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"]
Nov 01 09:14:38.496317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.496282    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b893c8-5m8ph"]
Nov 01 09:14:38.742588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.742481    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:38.928907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:38.928874    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87627620-7882-4622-8f09-8aeae4d9c05f" path="/var/lib/kubelet/pods/87627620-7882-4622-8f09-8aeae4d9c05f/volumes"
Nov 01 09:14:39.457143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:39.457111    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:39.735201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:39.735170    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:39.882000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:39.881971    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:40.137527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:40.137434    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:40.481855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:40.481827    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:40.658432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:40.658394    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:41.288058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.287992    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 09:14:41.322096 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:41.322062    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:14:41.338470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.338442    4314 generic.go:356] "Generic (PLEG): container finished" podID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerID="fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4" exitCode=0
Nov 01 09:14:41.338626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.338493    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"
Nov 01 09:14:41.338626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.338523    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" event={"ID":"4fd04ace-2609-40d0-9a61-82b45be1f283","Type":"ContainerDied","Data":"fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4"}
Nov 01 09:14:41.338626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.338590    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj" event={"ID":"4fd04ace-2609-40d0-9a61-82b45be1f283","Type":"ContainerDied","Data":"1f92b2df3a4fd821adac1ca1cf008940525f7d75541f91d8e1ea32f0f470ca80"}
Nov 01 09:14:41.338626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.338614    4314 scope.go:122] "RemoveContainer" containerID="fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4"
Nov 01 09:14:41.348099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.348081    4314 scope.go:122] "RemoveContainer" containerID="75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"
Nov 01 09:14:41.356782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.356754    4314 scope.go:122] "RemoveContainer" containerID="fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4"
Nov 01 09:14:41.357049 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:41.357032    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4\": container with ID starting with fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4 not found: ID does not exist" containerID="fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4"
Nov 01 09:14:41.357089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.357058    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4"} err="failed to get container status \"fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4\": rpc error: code = NotFound desc = could not find container \"fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4\": container with ID starting with fac669e24a42b11b218c0fcee65d8a530d2d403357aec2d4473fbc36269478a4 not found: ID does not exist"
Nov 01 09:14:41.357089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.357076    4314 scope.go:122] "RemoveContainer" containerID="75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"
Nov 01 09:14:41.357296 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:14:41.357277    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1\": container with ID starting with 75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1 not found: ID does not exist" containerID="75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"
Nov 01 09:14:41.357342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.357303    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1"} err="failed to get container status \"75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1\": rpc error: code = NotFound desc = could not find container \"75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1\": container with ID starting with 75bc97e129bd32cfb570d1ec3e4dea75bd06d0e537d3980ded1278144404c8a1 not found: ID does not exist"
Nov 01 09:14:41.364337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364317    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364354    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364386    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364498    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364588    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364640    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.364711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364678    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364709    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:41.364764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364731    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364772    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364839    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.364978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364888    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy\") pod \"4fd04ace-2609-40d0-9a61-82b45be1f283\" (UID: \"4fd04ace-2609-40d0-9a61-82b45be1f283\") "
Nov 01 09:14:41.364978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.364939    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.365123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.365101    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.365427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.365405    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.365508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.365434    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.365508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.365450    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd04ace-2609-40d0-9a61-82b45be1f283-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.365508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.365466    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.365508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.365480    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.366565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.366525    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "kube-api-access-s2q2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:41.366704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.366681    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:41.366914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.366895    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:14:41.367033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.367017    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy" pod "4fd04ace-2609-40d0-9a61-82b45be1f283" (UID: "4fd04ace-2609-40d0-9a61-82b45be1f283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.467109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.467072    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.467109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.467106    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2q2n\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-kube-api-access-s2q2n\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.467109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.467116    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.467342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.467128    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd04ace-2609-40d0-9a61-82b45be1f283-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.755007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.754973    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"]
Nov 01 09:14:41.759073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:41.759045    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bhtzj"]
Nov 01 09:14:42.928410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:42.928376    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" path="/var/lib/kubelet/pods/4fd04ace-2609-40d0-9a61-82b45be1f283/volumes"
Nov 01 09:14:43.286315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.286287    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:14:43.286558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.286362    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:14:43.357227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.357196    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:43.521214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.521151    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:43.764407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.764375    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"]
Nov 01 09:14:43.764880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.764865    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy"
Nov 01 09:14:43.764932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.764882    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy"
Nov 01 09:14:43.765003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.764994    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy"
Nov 01 09:14:43.765046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.765007    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy"
Nov 01 09:14:43.769846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.769829    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.790606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.790581    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.790735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.790696    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r75s\" (UniqueName: \"kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s\") pod \"payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.818966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.818931    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-514eb3-dockercfg-mcrrg\""
Nov 01 09:14:43.819121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.819010    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-514eb3\""
Nov 01 09:14:43.880585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.880551    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"]
Nov 01 09:14:43.892335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.892294    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.892495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.892407    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r75s\" (UniqueName: \"kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s\") pod \"payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.893035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.893015    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.926997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.926969    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"]
Nov 01 09:14:43.927478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.927465    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy"
Nov 01 09:14:43.927525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.927480    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd04ace-2609-40d0-9a61-82b45be1f283" containerName="istio-proxy"
Nov 01 09:14:43.931094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.931078    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:43.949745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.949721    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r75s\" (UniqueName: \"kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s\") pod \"payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:43.984934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.984900    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"]
Nov 01 09:14:43.993246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.993220    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:43.993367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:43.993281    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2qj\" (UniqueName: \"kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj\") pod \"payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:44.083058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.082981    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:44.094678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.094641    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:44.094786 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.094746    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2qj\" (UniqueName: \"kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj\") pod \"payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:44.095488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.095463    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:44.153842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.153813    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2qj\" (UniqueName: \"kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj\") pod \"payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:44.244981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.244950    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:44.452863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.452837    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"]
Nov 01 09:14:44.453418 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:14:44.453392    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fa2b66_90c1_4f83_bcb2_82e9e9e63b41.slice/crio-11841662b13967a5bef9a4a8a8bd916a09ce4322f49e83d9d3ff199b660de165 WatchSource:0}: Error finding container 11841662b13967a5bef9a4a8a8bd916a09ce4322f49e83d9d3ff199b660de165: Status 404 returned error can't find the container with id 11841662b13967a5bef9a4a8a8bd916a09ce4322f49e83d9d3ff199b660de165
Nov 01 09:14:44.619942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:44.619915    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"]
Nov 01 09:14:44.621090 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:14:44.621066    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b24a2a_9405_442d_a4ef_afa830ce8aab.slice/crio-b53c1cc4fcd16fb0a88077ad131848510a23a66c363d4d58890b6d912b9a032c WatchSource:0}: Error finding container b53c1cc4fcd16fb0a88077ad131848510a23a66c363d4d58890b6d912b9a032c: Status 404 returned error can't find the container with id b53c1cc4fcd16fb0a88077ad131848510a23a66c363d4d58890b6d912b9a032c
Nov 01 09:14:45.369328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.368468    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz" event={"ID":"61b24a2a-9405-442d-a4ef-afa830ce8aab","Type":"ContainerStarted","Data":"18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c"}
Nov 01 09:14:45.369328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.368596    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz" event={"ID":"61b24a2a-9405-442d-a4ef-afa830ce8aab","Type":"ContainerStarted","Data":"b53c1cc4fcd16fb0a88077ad131848510a23a66c363d4d58890b6d912b9a032c"}
Nov 01 09:14:45.369328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.368812    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:45.370385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.370357    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf" event={"ID":"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41","Type":"ContainerStarted","Data":"3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b"}
Nov 01 09:14:45.370512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.370402    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf" event={"ID":"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41","Type":"ContainerStarted","Data":"11841662b13967a5bef9a4a8a8bd916a09ce4322f49e83d9d3ff199b660de165"}
Nov 01 09:14:45.370512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.370454    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:45.477218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.477148    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz" podStartSLOduration=2.477132282 podStartE2EDuration="2.477132282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:14:45.476604653 +0000 UTC m=+2558.670999208" watchObservedRunningTime="2026-11-01 09:14:45.477132282 +0000 UTC m=+2558.671526838"
Nov 01 09:14:45.589570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.589498    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf" podStartSLOduration=2.589483376 podStartE2EDuration="2.589483376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:14:45.586998811 +0000 UTC m=+2558.781393367" watchObservedRunningTime="2026-11-01 09:14:45.589483376 +0000 UTC m=+2558.783877931"
Nov 01 09:14:45.641621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:45.641517    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:46.960485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:46.960451    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:47.359056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:47.359021    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:47.509171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:47.509140    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:48.916877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:48.916799    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:49.420587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:49.420527    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:50.163411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:50.163376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:52.190761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:52.190728    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:54.316370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:54.316336    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:54.465919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:54.465890    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:55.709492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:55.709407    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:55.857767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:55.857726    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:56.221822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:56.221792    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:56.378588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:56.378558    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:14:56.378758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:56.378636    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:14:56.729981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:56.729946    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:14:58.728448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:14:58.728403    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:00.162700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.162662    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"]
Nov 01 09:15:00.167754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.167727    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.168569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.168526    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"]
Nov 01 09:15:00.172243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.172225    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"]
Nov 01 09:15:00.172344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.172332    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.176091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.176074    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.208779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.208755    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"]
Nov 01 09:15:00.212559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.212522    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"]
Nov 01 09:15:00.212713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.212609    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.216687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.216668    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"]
Nov 01 09:15:00.216797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.216766    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.219361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.219337    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:15:00.219472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.219413    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:15:00.219591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.219473    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:15:00.220937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.220817    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.260420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.260384    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"]
Nov 01 09:15:00.265502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.265470    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"]
Nov 01 09:15:00.268751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.268724    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"]
Nov 01 09:15:00.271315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.271291    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"]
Nov 01 09:15:00.295785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.295754    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bld\" (UniqueName: \"kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.296108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.296090    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.296210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.296161    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.296314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.296278    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrj9\" (UniqueName: \"kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.296451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.296356    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szl67\" (UniqueName: \"kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.296451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.296414    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.306865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.306833    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"]
Nov 01 09:15:00.368522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.368492    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"]
Nov 01 09:15:00.398033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.397995    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkl6s\" (UniqueName: \"kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.398177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398045    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.398177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398115    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmpv\" (UniqueName: \"kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv\") pod \"maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398194    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398265    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kph2\" (UniqueName: \"kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2\") pod \"maas-api-key-cleanup-29892075-jt62g\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398311    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrj9\" (UniqueName: \"kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398465    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398507    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szl67\" (UniqueName: \"kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398643    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892075-jt62g\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.399089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398731    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.399089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398831    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bld\" (UniqueName: \"kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.399089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.398906    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.399089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.399085    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.399327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.399128    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.399487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.399466    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.455882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.455820    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrj9\" (UniqueName: \"kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.456003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.455878    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szl67\" (UniqueName: \"kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.456003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.455992    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bld\" (UniqueName: \"kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.483684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.483656    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:00.491138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.491113    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:00.497202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.497175    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:00.499800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.499777    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.499913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.499833    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkl6s\" (UniqueName: \"kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.499913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.499868    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmpv\" (UniqueName: \"kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv\") pod \"maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.500028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.499948    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kph2\" (UniqueName: \"kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2\") pod \"maas-api-key-cleanup-29892075-jt62g\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.500028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.499993    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.500138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.500029    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892075-jt62g\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.500572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.500547    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.500832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.500807    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892075-jt62g\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.501039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.501021    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.559315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.559258    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkl6s\" (UniqueName: \"kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.559718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.559697    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmpv\" (UniqueName: \"kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv\") pod \"maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.561881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.561117    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kph2\" (UniqueName: \"kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2\") pod \"maas-api-key-cleanup-29892075-jt62g\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.827855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.827820    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:00.835573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.835546    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:00.841687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.841664    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:00.875710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.875684    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"]
Nov 01 09:15:00.945254 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:00.945216    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587ed1ef_d8e9_4bc9_b2a4_4ea5e8810c0c.slice/crio-c3c3cd7641cf764c1a4ae45688be9b4609b0e4823e116a33648e1f2bb903cef6 WatchSource:0}: Error finding container c3c3cd7641cf764c1a4ae45688be9b4609b0e4823e116a33648e1f2bb903cef6: Status 404 returned error can't find the container with id c3c3cd7641cf764c1a4ae45688be9b4609b0e4823e116a33648e1f2bb903cef6
Nov 01 09:15:00.947733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.947709    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"]
Nov 01 09:15:00.947841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:00.947773    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"]
Nov 01 09:15:01.217545 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:01.217499    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab5295f_74d1_4228_a096_147d5ee3e22b.slice/crio-08c9330510ac5d551c8dc29263bdf6efeff0600c074d237de4e6c91f472a42b6 WatchSource:0}: Error finding container 08c9330510ac5d551c8dc29263bdf6efeff0600c074d237de4e6c91f472a42b6: Status 404 returned error can't find the container with id 08c9330510ac5d551c8dc29263bdf6efeff0600c074d237de4e6c91f472a42b6
Nov 01 09:15:01.217545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.217515    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"]
Nov 01 09:15:01.222063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.222034    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"]
Nov 01 09:15:01.223115 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:01.223089    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ed21be_6eea_4ed5_95b7_d889d944d47b.slice/crio-b3214c773805653a3af1bd18451d469ad97b88abbd551a625ed2985e6b2c74f7 WatchSource:0}: Error finding container b3214c773805653a3af1bd18451d469ad97b88abbd551a625ed2985e6b2c74f7: Status 404 returned error can't find the container with id b3214c773805653a3af1bd18451d469ad97b88abbd551a625ed2985e6b2c74f7
Nov 01 09:15:01.229487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.229458    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"]
Nov 01 09:15:01.231572 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:01.231515    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19b1aae_3c42_4587_be3d_ed43e26b2de1.slice/crio-b8672734bb2b4bc83d16e2cca79f842ba3dc330313321fdcbb390e5a6d2315b9 WatchSource:0}: Error finding container b8672734bb2b4bc83d16e2cca79f842ba3dc330313321fdcbb390e5a6d2315b9: Status 404 returned error can't find the container with id b8672734bb2b4bc83d16e2cca79f842ba3dc330313321fdcbb390e5a6d2315b9
Nov 01 09:15:01.481102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.481065    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerStarted","Data":"7540b0dd86b72779385f4eb9f6d5fea0601cc18ad17b93f67ba680246d038c9e"}
Nov 01 09:15:01.481270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.481152    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerStarted","Data":"c3c3cd7641cf764c1a4ae45688be9b4609b0e4823e116a33648e1f2bb903cef6"}
Nov 01 09:15:01.482741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.482712    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerStarted","Data":"bca1ec46ef3a8bda051f99e99fdbc696af9b52e998a2fd8d662b28b201d2dd47"}
Nov 01 09:15:01.482882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.482760    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerStarted","Data":"f74bcd07c4dfe644027eb47dc965a4da10fad4f02f1238a52a3b5f10f41a877c"}
Nov 01 09:15:01.484092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.484063    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerStarted","Data":"b3214c773805653a3af1bd18451d469ad97b88abbd551a625ed2985e6b2c74f7"}
Nov 01 09:15:01.485300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.485278    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" event={"ID":"a19b1aae-3c42-4587-be3d-ed43e26b2de1","Type":"ContainerStarted","Data":"b8672734bb2b4bc83d16e2cca79f842ba3dc330313321fdcbb390e5a6d2315b9"}
Nov 01 09:15:01.486500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.486475    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerStarted","Data":"08c9330510ac5d551c8dc29263bdf6efeff0600c074d237de4e6c91f472a42b6"}
Nov 01 09:15:01.487983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.487960    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerStarted","Data":"a3d6a5c2e27488881c53c355e8956f738ac1b80de306d3e17631b5d3090baba1"}
Nov 01 09:15:01.488093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.487996    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerStarted","Data":"7e16bdd8915c577021017e1fd27ae88be9518d9cf00c6d84cd22e3e82298560f"}
Nov 01 09:15:01.589617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.589568    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" podStartSLOduration=1.589554733 podStartE2EDuration="1.589554733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:01.587964868 +0000 UTC m=+2574.782359424" watchObservedRunningTime="2026-11-01 09:15:01.589554733 +0000 UTC m=+2574.783949285"
Nov 01 09:15:01.697518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.697473    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" podStartSLOduration=1.69745894 podStartE2EDuration="1.69745894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:01.694715879 +0000 UTC m=+2574.889110437" watchObservedRunningTime="2026-11-01 09:15:01.69745894 +0000 UTC m=+2574.891853507"
Nov 01 09:15:01.803432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:01.803375    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" podStartSLOduration=1.8033583819999999 podStartE2EDuration="1.803358382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:01.800973617 +0000 UTC m=+2574.995368171" watchObservedRunningTime="2026-11-01 09:15:01.803358382 +0000 UTC m=+2574.997752937"
Nov 01 09:15:02.426101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.426071    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:02.496245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.496213    4314 generic.go:356] "Generic (PLEG): container finished" podID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerID="c5ff913f481021f2cdd038a7a9aa9a360c341c0fd37216bc7dc1bc774ed8eca5" exitCode=60
Nov 01 09:15:02.496400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.496298    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerDied","Data":"c5ff913f481021f2cdd038a7a9aa9a360c341c0fd37216bc7dc1bc774ed8eca5"}
Nov 01 09:15:02.496636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.496620    4314 scope.go:122] "RemoveContainer" containerID="c5ff913f481021f2cdd038a7a9aa9a360c341c0fd37216bc7dc1bc774ed8eca5"
Nov 01 09:15:02.497983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.497965    4314 generic.go:356] "Generic (PLEG): container finished" podID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerID="a3d6a5c2e27488881c53c355e8956f738ac1b80de306d3e17631b5d3090baba1" exitCode=60
Nov 01 09:15:02.498080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.498050    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerDied","Data":"a3d6a5c2e27488881c53c355e8956f738ac1b80de306d3e17631b5d3090baba1"}
Nov 01 09:15:02.498294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.498281    4314 scope.go:122] "RemoveContainer" containerID="a3d6a5c2e27488881c53c355e8956f738ac1b80de306d3e17631b5d3090baba1"
Nov 01 09:15:02.499907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.499820    4314 generic.go:356] "Generic (PLEG): container finished" podID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerID="7540b0dd86b72779385f4eb9f6d5fea0601cc18ad17b93f67ba680246d038c9e" exitCode=60
Nov 01 09:15:02.499907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.499890    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerDied","Data":"7540b0dd86b72779385f4eb9f6d5fea0601cc18ad17b93f67ba680246d038c9e"}
Nov 01 09:15:02.500162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.500142    4314 scope.go:122] "RemoveContainer" containerID="7540b0dd86b72779385f4eb9f6d5fea0601cc18ad17b93f67ba680246d038c9e"
Nov 01 09:15:02.501527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.501508    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerID="bca1ec46ef3a8bda051f99e99fdbc696af9b52e998a2fd8d662b28b201d2dd47" exitCode=60
Nov 01 09:15:02.501599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.501559    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerDied","Data":"bca1ec46ef3a8bda051f99e99fdbc696af9b52e998a2fd8d662b28b201d2dd47"}
Nov 01 09:15:02.501784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.501770    4314 scope.go:122] "RemoveContainer" containerID="bca1ec46ef3a8bda051f99e99fdbc696af9b52e998a2fd8d662b28b201d2dd47"
Nov 01 09:15:02.503184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.503168    4314 generic.go:356] "Generic (PLEG): container finished" podID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerID="6155ca5bc28a5f7366dfdbf17db413152e6dd20475cf88244be7a705c9175dc0" exitCode=60
Nov 01 09:15:02.503260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.503223    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerDied","Data":"6155ca5bc28a5f7366dfdbf17db413152e6dd20475cf88244be7a705c9175dc0"}
Nov 01 09:15:02.503450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.503436    4314 scope.go:122] "RemoveContainer" containerID="6155ca5bc28a5f7366dfdbf17db413152e6dd20475cf88244be7a705c9175dc0"
Nov 01 09:15:02.504837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.504818    4314 generic.go:356] "Generic (PLEG): container finished" podID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerID="233920f5121f7901bb3ad2a56a585dbfb309dacf2998a16cd3bfd558cd341fbd" exitCode=60
Nov 01 09:15:02.504918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.504858    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" event={"ID":"a19b1aae-3c42-4587-be3d-ed43e26b2de1","Type":"ContainerDied","Data":"233920f5121f7901bb3ad2a56a585dbfb309dacf2998a16cd3bfd558cd341fbd"}
Nov 01 09:15:02.505072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:02.505058    4314 scope.go:122] "RemoveContainer" containerID="233920f5121f7901bb3ad2a56a585dbfb309dacf2998a16cd3bfd558cd341fbd"
Nov 01 09:15:03.514224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.514188    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerStarted","Data":"bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3"}
Nov 01 09:15:03.516395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.516367    4314 generic.go:356] "Generic (PLEG): container finished" podID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerID="abef4cce7446209c67e66845a13a0e62075092647a60152345f4eb21561da47b" exitCode=60
Nov 01 09:15:03.516578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.516462    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerDied","Data":"abef4cce7446209c67e66845a13a0e62075092647a60152345f4eb21561da47b"}
Nov 01 09:15:03.516578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.516527    4314 scope.go:122] "RemoveContainer" containerID="7540b0dd86b72779385f4eb9f6d5fea0601cc18ad17b93f67ba680246d038c9e"
Nov 01 09:15:03.516851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.516818    4314 scope.go:122] "RemoveContainer" containerID="abef4cce7446209c67e66845a13a0e62075092647a60152345f4eb21561da47b"
Nov 01 09:15:03.517174 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:03.517150    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb_odh-ai-gateway-infra(587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c"
Nov 01 09:15:03.518582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.518559    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerStarted","Data":"d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012"}
Nov 01 09:15:03.520509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.520487    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerStarted","Data":"6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535"}
Nov 01 09:15:03.522588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.522564    4314 generic.go:356] "Generic (PLEG): container finished" podID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerID="8c8cc4e053da95e093386fc91ae2449531046e320cb319996efeee3d9a918b32" exitCode=60
Nov 01 09:15:03.522690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.522639    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" event={"ID":"a19b1aae-3c42-4587-be3d-ed43e26b2de1","Type":"ContainerDied","Data":"8c8cc4e053da95e093386fc91ae2449531046e320cb319996efeee3d9a918b32"}
Nov 01 09:15:03.522897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.522879    4314 scope.go:122] "RemoveContainer" containerID="8c8cc4e053da95e093386fc91ae2449531046e320cb319996efeee3d9a918b32"
Nov 01 09:15:03.523167 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:03.523142    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892075-jt62g_odh-ai-gateway-infra(a19b1aae-3c42-4587-be3d-ed43e26b2de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1"
Nov 01 09:15:03.524791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.524757    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerStarted","Data":"b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3"}
Nov 01 09:15:03.641454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.641430    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:03.721584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.721562    4314 scope.go:122] "RemoveContainer" containerID="233920f5121f7901bb3ad2a56a585dbfb309dacf2998a16cd3bfd558cd341fbd"
Nov 01 09:15:03.789927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.789891    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:03.841496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:03.841448    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" podStartSLOduration=3.841434283 podStartE2EDuration="3.841434283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:03.838071918 +0000 UTC m=+2577.032466474" watchObservedRunningTime="2026-11-01 09:15:03.841434283 +0000 UTC m=+2577.035828838"
Nov 01 09:15:04.037662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.037627    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:04.055511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.055461    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" podStartSLOduration=4.05544684 podStartE2EDuration="4.05544684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:04.051880442 +0000 UTC m=+2577.246275010" watchObservedRunningTime="2026-11-01 09:15:04.05544684 +0000 UTC m=+2577.249841392"
Nov 01 09:15:04.534130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.534088    4314 scope.go:122] "RemoveContainer" containerID="abef4cce7446209c67e66845a13a0e62075092647a60152345f4eb21561da47b"
Nov 01 09:15:04.534621 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:04.534417    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb_odh-ai-gateway-infra(587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c"
Nov 01 09:15:04.535666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.535638    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerID="d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012" exitCode=60
Nov 01 09:15:04.535793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.535720    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerDied","Data":"d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012"}
Nov 01 09:15:04.535847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.535803    4314 scope.go:122] "RemoveContainer" containerID="bca1ec46ef3a8bda051f99e99fdbc696af9b52e998a2fd8d662b28b201d2dd47"
Nov 01 09:15:04.535999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.535979    4314 scope.go:122] "RemoveContainer" containerID="d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012"
Nov 01 09:15:04.536280 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:04.536257    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc_odh-ai-gateway-infra(1f440dd5-92ad-4506-b7d2-d1d58a458b14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14"
Nov 01 09:15:04.537714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.537695    4314 generic.go:356] "Generic (PLEG): container finished" podID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerID="6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535" exitCode=60
Nov 01 09:15:04.537841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.537785    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerDied","Data":"6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535"}
Nov 01 09:15:04.538013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.537995    4314 scope.go:122] "RemoveContainer" containerID="6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535"
Nov 01 09:15:04.538282 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:04.538261    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt_odh-ai-gateway-infra(78ed21be-6eea-4ed5-95b7-d889d944d47b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b"
Nov 01 09:15:04.539511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.539492    4314 scope.go:122] "RemoveContainer" containerID="8c8cc4e053da95e093386fc91ae2449531046e320cb319996efeee3d9a918b32"
Nov 01 09:15:04.539783 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:04.539760    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892075-jt62g_odh-ai-gateway-infra(a19b1aae-3c42-4587-be3d-ed43e26b2de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1"
Nov 01 09:15:04.541114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.541093    4314 generic.go:356] "Generic (PLEG): container finished" podID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerID="b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3" exitCode=60
Nov 01 09:15:04.541214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.541154    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerDied","Data":"b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3"}
Nov 01 09:15:04.541369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.541352    4314 scope.go:122] "RemoveContainer" containerID="b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3"
Nov 01 09:15:04.541677 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:04.541651    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g_odh-ai-gateway-infra(dab5295f-74d1-4228-a096-147d5ee3e22b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b"
Nov 01 09:15:04.543141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.543124    4314 generic.go:356] "Generic (PLEG): container finished" podID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerID="bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3" exitCode=60
Nov 01 09:15:04.543232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.543187    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerDied","Data":"bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3"}
Nov 01 09:15:04.543458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.543437    4314 scope.go:122] "RemoveContainer" containerID="bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3"
Nov 01 09:15:04.543751 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:04.543731    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk_odh-ai-gateway-infra(fa4be9b3-a9e9-43a1-8148-1ea1813e2279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279"
Nov 01 09:15:04.549061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.549026    4314 scope.go:122] "RemoveContainer" containerID="6155ca5bc28a5f7366dfdbf17db413152e6dd20475cf88244be7a705c9175dc0"
Nov 01 09:15:04.561434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.561415    4314 scope.go:122] "RemoveContainer" containerID="c5ff913f481021f2cdd038a7a9aa9a360c341c0fd37216bc7dc1bc774ed8eca5"
Nov 01 09:15:04.573717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:04.573698    4314 scope.go:122] "RemoveContainer" containerID="a3d6a5c2e27488881c53c355e8956f738ac1b80de306d3e17631b5d3090baba1"
Nov 01 09:15:05.140463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.140430    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:05.550878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.550848    4314 scope.go:122] "RemoveContainer" containerID="bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3"
Nov 01 09:15:05.551298 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:05.551184    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk_odh-ai-gateway-infra(fa4be9b3-a9e9-43a1-8148-1ea1813e2279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279"
Nov 01 09:15:05.552474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.552449    4314 scope.go:122] "RemoveContainer" containerID="d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012"
Nov 01 09:15:05.552771 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:05.552751    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc_odh-ai-gateway-infra(1f440dd5-92ad-4506-b7d2-d1d58a458b14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14"
Nov 01 09:15:05.554055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.554035    4314 scope.go:122] "RemoveContainer" containerID="6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535"
Nov 01 09:15:05.554277 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:05.554253    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt_odh-ai-gateway-infra(78ed21be-6eea-4ed5-95b7-d889d944d47b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b"
Nov 01 09:15:05.555625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.555604    4314 scope.go:122] "RemoveContainer" containerID="b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3"
Nov 01 09:15:05.555831 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:05.555811    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g_odh-ai-gateway-infra(dab5295f-74d1-4228-a096-147d5ee3e22b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b"
Nov 01 09:15:05.791645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.791611    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:05.951469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:05.951347    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:09.218255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:09.218219    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:11.323261 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:11.323223    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:15:13.921702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:13.921667    4314 scope.go:122] "RemoveContainer" containerID="6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535"
Nov 01 09:15:14.187824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:14.187738    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:14.626105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:14.626070    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerStarted","Data":"85ece794d7e9fc3a00b8e2b1f88ac35011358cb2230b00cd42f76adf699c7f7e"}
Nov 01 09:15:14.659252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:14.659220    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:14.921939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:14.921857    4314 scope.go:122] "RemoveContainer" containerID="d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012"
Nov 01 09:15:15.634128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:15.634083    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerStarted","Data":"b3b3c352eb16a485a55202d06b408a29e67bbfaef19204337cf3bd6b26f7fe61"}
Nov 01 09:15:15.636098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:15.636072    4314 generic.go:356] "Generic (PLEG): container finished" podID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerID="85ece794d7e9fc3a00b8e2b1f88ac35011358cb2230b00cd42f76adf699c7f7e" exitCode=60
Nov 01 09:15:15.636247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:15.636146    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerDied","Data":"85ece794d7e9fc3a00b8e2b1f88ac35011358cb2230b00cd42f76adf699c7f7e"}
Nov 01 09:15:15.636247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:15.636207    4314 scope.go:122] "RemoveContainer" containerID="6d2460e4ce0355f7aa0607b4da4c3467d21bdeb6d598ebb72be65677ca8f1535"
Nov 01 09:15:15.637097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:15.636525    4314 scope.go:122] "RemoveContainer" containerID="85ece794d7e9fc3a00b8e2b1f88ac35011358cb2230b00cd42f76adf699c7f7e"
Nov 01 09:15:15.637097 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:15.636815    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt_odh-ai-gateway-infra(78ed21be-6eea-4ed5-95b7-d889d944d47b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b"
Nov 01 09:15:15.745258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:15.745228    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"]
Nov 01 09:15:16.644476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.644446    4314 generic.go:356] "Generic (PLEG): container finished" podID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerID="b3b3c352eb16a485a55202d06b408a29e67bbfaef19204337cf3bd6b26f7fe61" exitCode=60
Nov 01 09:15:16.644972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.644519    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerDied","Data":"b3b3c352eb16a485a55202d06b408a29e67bbfaef19204337cf3bd6b26f7fe61"}
Nov 01 09:15:16.644972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.644602    4314 scope.go:122] "RemoveContainer" containerID="d6e465a1f4b8c0e768947af2db5cede35d56eff3b0eec623ca43d3c8071a4012"
Nov 01 09:15:16.645085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.645014    4314 scope.go:122] "RemoveContainer" containerID="b3b3c352eb16a485a55202d06b408a29e67bbfaef19204337cf3bd6b26f7fe61"
Nov 01 09:15:16.645327 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:16.645302    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc_odh-ai-gateway-infra(1f440dd5-92ad-4506-b7d2-d1d58a458b14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14"
Nov 01 09:15:16.786209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.786184    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:16.802104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.802075    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"]
Nov 01 09:15:16.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.898859    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca\") pod \"78ed21be-6eea-4ed5-95b7-d889d944d47b\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") "
Nov 01 09:15:16.898944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.898931    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv\" (UniqueName: \"kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv\") pod \"78ed21be-6eea-4ed5-95b7-d889d944d47b\" (UID: \"78ed21be-6eea-4ed5-95b7-d889d944d47b\") "
Nov 01 09:15:16.899272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.899249    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca" pod "78ed21be-6eea-4ed5-95b7-d889d944d47b" (UID: "78ed21be-6eea-4ed5-95b7-d889d944d47b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:16.901038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.901018    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv" pod "78ed21be-6eea-4ed5-95b7-d889d944d47b" (UID: "78ed21be-6eea-4ed5-95b7-d889d944d47b"). InnerVolumeSpecName "kube-api-access-mhmpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:16.936376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.936328    4314 scope.go:122] "RemoveContainer" containerID="abef4cce7446209c67e66845a13a0e62075092647a60152345f4eb21561da47b"
Nov 01 09:15:16.936600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.936579    4314 scope.go:122] "RemoveContainer" containerID="b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3"
Nov 01 09:15:16.936713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:16.936695    4314 scope.go:122] "RemoveContainer" containerID="8c8cc4e053da95e093386fc91ae2449531046e320cb319996efeee3d9a918b32"
Nov 01 09:15:17.000985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.000752    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ed21be-6eea-4ed5-95b7-d889d944d47b-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:17.000985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.000780    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmpv\" (UniqueName: \"kubernetes.io/projected/78ed21be-6eea-4ed5-95b7-d889d944d47b-kube-api-access-mhmpv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:17.655163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.655126    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerStarted","Data":"92a257d1808a03c2e3ead68f06f76c2c783585945930d24428ce238c4e9b8429"}
Nov 01 09:15:17.658677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.658652    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"
Nov 01 09:15:17.659347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.659256    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt" event={"ID":"78ed21be-6eea-4ed5-95b7-d889d944d47b","Type":"ContainerDied","Data":"b3214c773805653a3af1bd18451d469ad97b88abbd551a625ed2985e6b2c74f7"}
Nov 01 09:15:17.659347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.659313    4314 scope.go:122] "RemoveContainer" containerID="85ece794d7e9fc3a00b8e2b1f88ac35011358cb2230b00cd42f76adf699c7f7e"
Nov 01 09:15:17.663087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.663064    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" event={"ID":"a19b1aae-3c42-4587-be3d-ed43e26b2de1","Type":"ContainerStarted","Data":"a93974731383f03f5a06f7e5518341883d02080822384c516be81619c8a44888"}
Nov 01 09:15:17.665391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.665368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerStarted","Data":"f36417d494d28e5d31e42cb5eb3e01ffb948fef2a07d7c3285e5b4b0f9f836c6"}
Nov 01 09:15:17.863473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.863445    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:17.912445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.912418    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca\") pod \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") "
Nov 01 09:15:17.912660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.912618    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld\" (UniqueName: \"kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld\") pod \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\" (UID: \"1f440dd5-92ad-4506-b7d2-d1d58a458b14\") "
Nov 01 09:15:17.912766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.912735    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca" pod "1f440dd5-92ad-4506-b7d2-d1d58a458b14" (UID: "1f440dd5-92ad-4506-b7d2-d1d58a458b14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:17.912967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.912953    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f440dd5-92ad-4506-b7d2-d1d58a458b14-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:17.921467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.921443    4314 scope.go:122] "RemoveContainer" containerID="bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3"
Nov 01 09:15:17.958286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:17.958035    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld" pod "1f440dd5-92ad-4506-b7d2-d1d58a458b14" (UID: "1f440dd5-92ad-4506-b7d2-d1d58a458b14"). InnerVolumeSpecName "kube-api-access-66bld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:18.014775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.014744    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bld\" (UniqueName: \"kubernetes.io/projected/1f440dd5-92ad-4506-b7d2-d1d58a458b14-kube-api-access-66bld\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:18.089856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.089810    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" podStartSLOduration=18.089796012 podStartE2EDuration="18.089796012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:18.087372292 +0000 UTC m=+2591.281766849" watchObservedRunningTime="2026-11-01 09:15:18.089796012 +0000 UTC m=+2591.284190566"
Nov 01 09:15:18.191861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.191832    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"]
Nov 01 09:15:18.198231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.198207    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-55644e-29892075-5kspt"]
Nov 01 09:15:18.673177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.673147    4314 generic.go:356] "Generic (PLEG): container finished" podID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerID="92a257d1808a03c2e3ead68f06f76c2c783585945930d24428ce238c4e9b8429" exitCode=60
Nov 01 09:15:18.673642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.673218    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerDied","Data":"92a257d1808a03c2e3ead68f06f76c2c783585945930d24428ce238c4e9b8429"}
Nov 01 09:15:18.673642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.673281    4314 scope.go:122] "RemoveContainer" containerID="abef4cce7446209c67e66845a13a0e62075092647a60152345f4eb21561da47b"
Nov 01 09:15:18.673770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.673681    4314 scope.go:122] "RemoveContainer" containerID="92a257d1808a03c2e3ead68f06f76c2c783585945930d24428ce238c4e9b8429"
Nov 01 09:15:18.674029 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:18.674008    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb_odh-ai-gateway-infra(587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c"
Nov 01 09:15:18.674918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.674892    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"
Nov 01 09:15:18.675042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.674894    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc" event={"ID":"1f440dd5-92ad-4506-b7d2-d1d58a458b14","Type":"ContainerDied","Data":"f74bcd07c4dfe644027eb47dc965a4da10fad4f02f1238a52a3b5f10f41a877c"}
Nov 01 09:15:18.677952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.677928    4314 generic.go:356] "Generic (PLEG): container finished" podID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerID="a93974731383f03f5a06f7e5518341883d02080822384c516be81619c8a44888" exitCode=60
Nov 01 09:15:18.678059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.677951    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" event={"ID":"a19b1aae-3c42-4587-be3d-ed43e26b2de1","Type":"ContainerDied","Data":"a93974731383f03f5a06f7e5518341883d02080822384c516be81619c8a44888"}
Nov 01 09:15:18.678256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.678242    4314 scope.go:122] "RemoveContainer" containerID="a93974731383f03f5a06f7e5518341883d02080822384c516be81619c8a44888"
Nov 01 09:15:18.678610 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:18.678518    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892075-jt62g_odh-ai-gateway-infra(a19b1aae-3c42-4587-be3d-ed43e26b2de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1"
Nov 01 09:15:18.680282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.680260    4314 generic.go:356] "Generic (PLEG): container finished" podID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerID="f36417d494d28e5d31e42cb5eb3e01ffb948fef2a07d7c3285e5b4b0f9f836c6" exitCode=60
Nov 01 09:15:18.680404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.680286    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerDied","Data":"f36417d494d28e5d31e42cb5eb3e01ffb948fef2a07d7c3285e5b4b0f9f836c6"}
Nov 01 09:15:18.680703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.680683    4314 scope.go:122] "RemoveContainer" containerID="f36417d494d28e5d31e42cb5eb3e01ffb948fef2a07d7c3285e5b4b0f9f836c6"
Nov 01 09:15:18.680999 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:18.680974    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g_odh-ai-gateway-infra(dab5295f-74d1-4228-a096-147d5ee3e22b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b"
Nov 01 09:15:18.682242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.682224    4314 generic.go:356] "Generic (PLEG): container finished" podID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerID="340526defb5eddf2843d25899cb829bbcff363b6276a6153d23aae68dba566bd" exitCode=60
Nov 01 09:15:18.682332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.682291    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerDied","Data":"340526defb5eddf2843d25899cb829bbcff363b6276a6153d23aae68dba566bd"}
Nov 01 09:15:18.682609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.682595    4314 scope.go:122] "RemoveContainer" containerID="340526defb5eddf2843d25899cb829bbcff363b6276a6153d23aae68dba566bd"
Nov 01 09:15:18.682833 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:18.682817    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk_odh-ai-gateway-infra(fa4be9b3-a9e9-43a1-8148-1ea1813e2279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279"
Nov 01 09:15:18.705069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.705052    4314 scope.go:122] "RemoveContainer" containerID="b3b3c352eb16a485a55202d06b408a29e67bbfaef19204337cf3bd6b26f7fe61"
Nov 01 09:15:18.713479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.713456    4314 scope.go:122] "RemoveContainer" containerID="8c8cc4e053da95e093386fc91ae2449531046e320cb319996efeee3d9a918b32"
Nov 01 09:15:18.726155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.726139    4314 scope.go:122] "RemoveContainer" containerID="b71b94989007c19a515f3af8e4bd9479cb0ceb9edbb7be61fa074320d7588da3"
Nov 01 09:15:18.738012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.737991    4314 scope.go:122] "RemoveContainer" containerID="bab88548a33ec680179063e9a13638a64e03d31a50869bc08223bcbb6ab5a3f3"
Nov 01 09:15:18.773049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.773015    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"]
Nov 01 09:15:18.929831 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.929755    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" path="/var/lib/kubelet/pods/78ed21be-6eea-4ed5-95b7-d889d944d47b/volumes"
Nov 01 09:15:18.988917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.988879    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:18.991499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:18.991478    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"]
Nov 01 09:15:19.097698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:19.097665    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"]
Nov 01 09:15:19.110457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:19.110418    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"]
Nov 01 09:15:19.111622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:19.111600    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be7cc-29892075-m4nmc"]
Nov 01 09:15:19.885152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:19.885125    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:19.908708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:19.908684    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:19.911891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:19.911871    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:20.038215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038182    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca\") pod \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") "
Nov 01 09:15:20.038389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038228    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca\") pod \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") "
Nov 01 09:15:20.038389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038327    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67\" (UniqueName: \"kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67\") pod \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\" (UID: \"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c\") "
Nov 01 09:15:20.038389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038365    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s\" (UniqueName: \"kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s\") pod \"dab5295f-74d1-4228-a096-147d5ee3e22b\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") "
Nov 01 09:15:20.038526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038440    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2\" (UniqueName: \"kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2\") pod \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\" (UID: \"a19b1aae-3c42-4587-be3d-ed43e26b2de1\") "
Nov 01 09:15:20.038621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038588    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca\") pod \"dab5295f-74d1-4228-a096-147d5ee3e22b\" (UID: \"dab5295f-74d1-4228-a096-147d5ee3e22b\") "
Nov 01 09:15:20.038621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038603    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca" pod "587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" (UID: "587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:20.038737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.038679    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca" pod "a19b1aae-3c42-4587-be3d-ed43e26b2de1" (UID: "a19b1aae-3c42-4587-be3d-ed43e26b2de1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:20.039091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.039028    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca" pod "dab5295f-74d1-4228-a096-147d5ee3e22b" (UID: "dab5295f-74d1-4228-a096-147d5ee3e22b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:20.039228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.039135    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab5295f-74d1-4228-a096-147d5ee3e22b-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.039228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.039149    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.039228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.039158    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b1aae-3c42-4587-be3d-ed43e26b2de1-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.040741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.040716    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67" pod "587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" (UID: "587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c"). InnerVolumeSpecName "kube-api-access-szl67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:20.041047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.041028    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s" pod "dab5295f-74d1-4228-a096-147d5ee3e22b" (UID: "dab5295f-74d1-4228-a096-147d5ee3e22b"). InnerVolumeSpecName "kube-api-access-fkl6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:20.041098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.041074    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2" pod "a19b1aae-3c42-4587-be3d-ed43e26b2de1" (UID: "a19b1aae-3c42-4587-be3d-ed43e26b2de1"). InnerVolumeSpecName "kube-api-access-5kph2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:20.140187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.140154    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szl67\" (UniqueName: \"kubernetes.io/projected/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c-kube-api-access-szl67\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.140187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.140184    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkl6s\" (UniqueName: \"kubernetes.io/projected/dab5295f-74d1-4228-a096-147d5ee3e22b-kube-api-access-fkl6s\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.140404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.140196    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kph2\" (UniqueName: \"kubernetes.io/projected/a19b1aae-3c42-4587-be3d-ed43e26b2de1-kube-api-access-5kph2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.225495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.225460    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"]
Nov 01 09:15:20.361853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.361828    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:20.443791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.443753    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca\") pod \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") "
Nov 01 09:15:20.443972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.443797    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9\" (UniqueName: \"kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9\") pod \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\" (UID: \"fa4be9b3-a9e9-43a1-8148-1ea1813e2279\") "
Nov 01 09:15:20.444100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.444078    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca" pod "fa4be9b3-a9e9-43a1-8148-1ea1813e2279" (UID: "fa4be9b3-a9e9-43a1-8148-1ea1813e2279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:20.444410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.444389    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.445968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.445945    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9" pod "fa4be9b3-a9e9-43a1-8148-1ea1813e2279" (UID: "fa4be9b3-a9e9-43a1-8148-1ea1813e2279"). InnerVolumeSpecName "kube-api-access-9xrj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:20.546122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.546092    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xrj9\" (UniqueName: \"kubernetes.io/projected/fa4be9b3-a9e9-43a1-8148-1ea1813e2279-kube-api-access-9xrj9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:20.705294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.705214    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"
Nov 01 09:15:20.705294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.705258    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb" event={"ID":"587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c","Type":"ContainerDied","Data":"c3c3cd7641cf764c1a4ae45688be9b4609b0e4823e116a33648e1f2bb903cef6"}
Nov 01 09:15:20.705581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.705319    4314 scope.go:122] "RemoveContainer" containerID="92a257d1808a03c2e3ead68f06f76c2c783585945930d24428ce238c4e9b8429"
Nov 01 09:15:20.706893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.706867    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"
Nov 01 09:15:20.706893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.706882    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g" event={"ID":"a19b1aae-3c42-4587-be3d-ed43e26b2de1","Type":"ContainerDied","Data":"b8672734bb2b4bc83d16e2cca79f842ba3dc330313321fdcbb390e5a6d2315b9"}
Nov 01 09:15:20.708507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.708481    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"
Nov 01 09:15:20.708638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.708481    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g" event={"ID":"dab5295f-74d1-4228-a096-147d5ee3e22b","Type":"ContainerDied","Data":"08c9330510ac5d551c8dc29263bdf6efeff0600c074d237de4e6c91f472a42b6"}
Nov 01 09:15:20.710164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.710144    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk" event={"ID":"fa4be9b3-a9e9-43a1-8148-1ea1813e2279","Type":"ContainerDied","Data":"7e16bdd8915c577021017e1fd27ae88be9518d9cf00c6d84cd22e3e82298560f"}
Nov 01 09:15:20.710293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.710228    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"
Nov 01 09:15:20.717923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.717905    4314 scope.go:122] "RemoveContainer" containerID="a93974731383f03f5a06f7e5518341883d02080822384c516be81619c8a44888"
Nov 01 09:15:20.727648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.727634    4314 scope.go:122] "RemoveContainer" containerID="f36417d494d28e5d31e42cb5eb3e01ffb948fef2a07d7c3285e5b4b0f9f836c6"
Nov 01 09:15:20.739514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.739493    4314 scope.go:122] "RemoveContainer" containerID="340526defb5eddf2843d25899cb829bbcff363b6276a6153d23aae68dba566bd"
Nov 01 09:15:20.788419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.788390    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"]
Nov 01 09:15:20.788980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.788961    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.788981    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.788995    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789002    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789019    4314 cpu_manager.go:414] "containerMap: removing container" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789026    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789034    4314 cpu_manager.go:414] "containerMap: removing container" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789042    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789061    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789068    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789079    4314 cpu_manager.go:414] "containerMap: removing container" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789086    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789102    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789108    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789120    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789127    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789149    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789156    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789183    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789191    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789205    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789212    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789224    4314 cpu_manager.go:414] "containerMap: removing container" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789231    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789243    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789250    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789382    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789394    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789404    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789412    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789421    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789433    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789443    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789453    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789462    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789471    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789480    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789490    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789499    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789506    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:20.789793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.789514    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:20.794827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.794809    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:20.843905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.843654    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:15:20.843905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.843773    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:15:20.843905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.843842    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:15:20.845220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.845187    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:15:20.846792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.846680    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"]
Nov 01 09:15:20.858704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.858679    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"]
Nov 01 09:15:20.864410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.864387    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892075-ms59g"]
Nov 01 09:15:20.929111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.929080    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" path="/var/lib/kubelet/pods/1f440dd5-92ad-4506-b7d2-d1d58a458b14/volumes"
Nov 01 09:15:20.929517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.929503    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" path="/var/lib/kubelet/pods/dab5295f-74d1-4228-a096-147d5ee3e22b/volumes"
Nov 01 09:15:20.950437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.950396    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:20.950605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.950449    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6q8l\" (UniqueName: \"kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:20.950657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.950625    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:20.968405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.968328    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"]
Nov 01 09:15:20.976709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:20.976682    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892075-jt62g"]
Nov 01 09:15:21.052100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.052059    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.052296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.052229    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.052296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.052291    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6q8l\" (UniqueName: \"kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.052894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.052874    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.054801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.054780    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.083382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.083352    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"]
Nov 01 09:15:21.087669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.087648    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892075-9b7rk"]
Nov 01 09:15:21.110312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.110279    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6q8l\" (UniqueName: \"kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l\") pod \"maas-api-revoke-sub-de8d1ab1-stplj\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.189851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.189820    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"]
Nov 01 09:15:21.196323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.196290    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892075-xc9cb"]
Nov 01 09:15:21.408577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.408519    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:21.778890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:21.778860    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"]
Nov 01 09:15:21.779954 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:21.779930    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e374f7_72a0_43ad_b1e9_2d20318e6735.slice/crio-5c50e9ddb5511bc8daa0dab21dbb0d8d84d97b4ff482ca8bce21135a2b243e9b WatchSource:0}: Error finding container 5c50e9ddb5511bc8daa0dab21dbb0d8d84d97b4ff482ca8bce21135a2b243e9b: Status 404 returned error can't find the container with id 5c50e9ddb5511bc8daa0dab21dbb0d8d84d97b4ff482ca8bce21135a2b243e9b
Nov 01 09:15:22.729666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:22.729635    4314 generic.go:356] "Generic (PLEG): container finished" podID="a3e374f7-72a0-43ad-b1e9-2d20318e6735" containerID="29f197fe442f3ffc036d7ddab162417fa2d91dc02cfa585802b84eef9363a4e8" exitCode=0
Nov 01 09:15:22.730063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:22.729723    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj" event={"ID":"a3e374f7-72a0-43ad-b1e9-2d20318e6735","Type":"ContainerDied","Data":"29f197fe442f3ffc036d7ddab162417fa2d91dc02cfa585802b84eef9363a4e8"}
Nov 01 09:15:22.730063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:22.729780    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj" event={"ID":"a3e374f7-72a0-43ad-b1e9-2d20318e6735","Type":"ContainerStarted","Data":"5c50e9ddb5511bc8daa0dab21dbb0d8d84d97b4ff482ca8bce21135a2b243e9b"}
Nov 01 09:15:22.928862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:22.928783    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" path="/var/lib/kubelet/pods/587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c/volumes"
Nov 01 09:15:22.929177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:22.929164    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19b1aae-3c42-4587-be3d-ed43e26b2de1" path="/var/lib/kubelet/pods/a19b1aae-3c42-4587-be3d-ed43e26b2de1/volumes"
Nov 01 09:15:22.929523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:22.929511    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" path="/var/lib/kubelet/pods/fa4be9b3-a9e9-43a1-8148-1ea1813e2279/volumes"
Nov 01 09:15:23.722973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.722928    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:23.888763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.888717    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:23.892733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.892711    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:23.987631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.987602    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca\") pod \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") "
Nov 01 09:15:23.987832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.987650    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l\" (UniqueName: \"kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l\") pod \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") "
Nov 01 09:15:23.987832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.987725    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca\") pod \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\" (UID: \"a3e374f7-72a0-43ad-b1e9-2d20318e6735\") "
Nov 01 09:15:23.988079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.988055    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca" pod "a3e374f7-72a0-43ad-b1e9-2d20318e6735" (UID: "a3e374f7-72a0-43ad-b1e9-2d20318e6735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:23.988197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.988184    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:23.989776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.989756    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca" pod "a3e374f7-72a0-43ad-b1e9-2d20318e6735" (UID: "a3e374f7-72a0-43ad-b1e9-2d20318e6735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:23.989862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:23.989778    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l" pod "a3e374f7-72a0-43ad-b1e9-2d20318e6735" (UID: "a3e374f7-72a0-43ad-b1e9-2d20318e6735"). InnerVolumeSpecName "kube-api-access-d6q8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:24.034805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:24.034770    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:24.089748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:24.089718    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e374f7-72a0-43ad-b1e9-2d20318e6735-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:24.089748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:24.089745    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6q8l\" (UniqueName: \"kubernetes.io/projected/a3e374f7-72a0-43ad-b1e9-2d20318e6735-kube-api-access-d6q8l\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:24.743526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:24.743497    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"
Nov 01 09:15:24.743526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:24.743509    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj" event={"ID":"a3e374f7-72a0-43ad-b1e9-2d20318e6735","Type":"ContainerDied","Data":"5c50e9ddb5511bc8daa0dab21dbb0d8d84d97b4ff482ca8bce21135a2b243e9b"}
Nov 01 09:15:24.743763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:24.743577    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c50e9ddb5511bc8daa0dab21dbb0d8d84d97b4ff482ca8bce21135a2b243e9b"
Nov 01 09:15:28.690916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:28.690885    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:33.584566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:33.584514    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:38.323126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.323095    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:38.487770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.487699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:38.685142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.685043    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:38.991524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.991494    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"]
Nov 01 09:15:38.992023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992010    4314 cpu_manager.go:414] "containerMap: removing container" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:38.992067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992025    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587ed1ef-d8e9-4bc9-b2a4-4ea5e8810c0c" containerName="cleanup"
Nov 01 09:15:38.992067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992047    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a3e374f7-72a0-43ad-b1e9-2d20318e6735" containerName="revoke-keys"
Nov 01 09:15:38.992067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992053    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e374f7-72a0-43ad-b1e9-2d20318e6735" containerName="revoke-keys"
Nov 01 09:15:38.992067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992064    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992069    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992078    4314 cpu_manager.go:414] "containerMap: removing container" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992082    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992090    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992095    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab5295f-74d1-4228-a096-147d5ee3e22b" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992108    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992113    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992185    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ed21be-6eea-4ed5-95b7-d889d944d47b" containerName="cleanup"
Nov 01 09:15:38.992196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992192    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4be9b3-a9e9-43a1-8148-1ea1813e2279" containerName="cleanup"
Nov 01 09:15:38.992552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992201    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e374f7-72a0-43ad-b1e9-2d20318e6735" containerName="revoke-keys"
Nov 01 09:15:38.992552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.992208    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f440dd5-92ad-4506-b7d2-d1d58a458b14" containerName="cleanup"
Nov 01 09:15:38.995662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:38.995644    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.045745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.045720    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:15:39.086482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.086440    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.086703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.086654    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.086703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.086689    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvd9\" (UniqueName: \"kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.091422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.091396    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:15:39.091608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.091458    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:15:39.091720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.091697    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:15:39.095489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.095467    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"]
Nov 01 09:15:39.187995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.187954    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.188184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.188050    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.188184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.188081    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvd9\" (UniqueName: \"kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.188770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.188746    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.190485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.190460    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.245761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.245695    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvd9\" (UniqueName: \"kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9\") pod \"maas-api-revoke-sub-6ef509b1-jwxrq\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.308868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.308826    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:39.696981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.696953    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"]
Nov 01 09:15:39.697844 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:39.697818    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce49ef46_8906_467f_b4d1_6898b4c3cba0.slice/crio-12915be7abb346e350a039e6c91de3cd68b4aa7e3c0fdc54501f6c13bdf545ba WatchSource:0}: Error finding container 12915be7abb346e350a039e6c91de3cd68b4aa7e3c0fdc54501f6c13bdf545ba: Status 404 returned error can't find the container with id 12915be7abb346e350a039e6c91de3cd68b4aa7e3c0fdc54501f6c13bdf545ba
Nov 01 09:15:39.838149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:39.838107    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq" event={"ID":"ce49ef46-8906-467f-b4d1-6898b4c3cba0","Type":"ContainerStarted","Data":"12915be7abb346e350a039e6c91de3cd68b4aa7e3c0fdc54501f6c13bdf545ba"}
Nov 01 09:15:40.847357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:40.847323    4314 generic.go:356] "Generic (PLEG): container finished" podID="ce49ef46-8906-467f-b4d1-6898b4c3cba0" containerID="d6fe76df9774520e59a62192e22ba2d38f5e417f5938dabc8bf6caa93102f9d3" exitCode=0
Nov 01 09:15:40.847777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:40.847405    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq" event={"ID":"ce49ef46-8906-467f-b4d1-6898b4c3cba0","Type":"ContainerDied","Data":"d6fe76df9774520e59a62192e22ba2d38f5e417f5938dabc8bf6caa93102f9d3"}
Nov 01 09:15:41.323391 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:41.323353    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:15:42.006145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.006114    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:42.132474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.132444    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9\" (UniqueName: \"kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9\") pod \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") "
Nov 01 09:15:42.132673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.132652    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca\") pod \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") "
Nov 01 09:15:42.132750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.132739    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca\") pod \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\" (UID: \"ce49ef46-8906-467f-b4d1-6898b4c3cba0\") "
Nov 01 09:15:42.133059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.133034    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca" pod "ce49ef46-8906-467f-b4d1-6898b4c3cba0" (UID: "ce49ef46-8906-467f-b4d1-6898b4c3cba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:42.133262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.133240    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:42.134750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.134727    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9" pod "ce49ef46-8906-467f-b4d1-6898b4c3cba0" (UID: "ce49ef46-8906-467f-b4d1-6898b4c3cba0"). InnerVolumeSpecName "kube-api-access-zvvd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:42.134955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.134930    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca" pod "ce49ef46-8906-467f-b4d1-6898b4c3cba0" (UID: "ce49ef46-8906-467f-b4d1-6898b4c3cba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:42.234969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.234934    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49ef46-8906-467f-b4d1-6898b4c3cba0-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:42.234969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.234963    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvd9\" (UniqueName: \"kubernetes.io/projected/ce49ef46-8906-467f-b4d1-6898b4c3cba0-kube-api-access-zvvd9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:42.660950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.660911    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:42.864550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.864508    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"
Nov 01 09:15:42.864740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.864572    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq" event={"ID":"ce49ef46-8906-467f-b4d1-6898b4c3cba0","Type":"ContainerDied","Data":"12915be7abb346e350a039e6c91de3cd68b4aa7e3c0fdc54501f6c13bdf545ba"}
Nov 01 09:15:42.864740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:42.864629    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12915be7abb346e350a039e6c91de3cd68b4aa7e3c0fdc54501f6c13bdf545ba"
Nov 01 09:15:43.072576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.072524    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"]
Nov 01 09:15:43.072971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.072791    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8" podUID="092a625d-0a4f-4790-95bb-c75ad7d0dc6f" containerName="payload-pre-processing" containerID="cri-o://618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b" gracePeriod=30
Nov 01 09:15:43.179725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.179637    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:43.330732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.330702    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:43.354169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.354143    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:15:43.450332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.450298    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm\" (UniqueName: \"kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm\") pod \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") "
Nov 01 09:15:43.450493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.450450    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume\") pod \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\" (UID: \"092a625d-0a4f-4790-95bb-c75ad7d0dc6f\") "
Nov 01 09:15:43.450889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.450858    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume" pod "092a625d-0a4f-4790-95bb-c75ad7d0dc6f" (UID: "092a625d-0a4f-4790-95bb-c75ad7d0dc6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:43.452481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.452455    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm" pod "092a625d-0a4f-4790-95bb-c75ad7d0dc6f" (UID: "092a625d-0a4f-4790-95bb-c75ad7d0dc6f"). InnerVolumeSpecName "kube-api-access-l4nsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:43.551810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.551753    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:43.552092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.552072    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4nsm\" (UniqueName: \"kubernetes.io/projected/092a625d-0a4f-4790-95bb-c75ad7d0dc6f-kube-api-access-l4nsm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:43.857959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.857928    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:43.871605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.871570    4314 generic.go:356] "Generic (PLEG): container finished" podID="092a625d-0a4f-4790-95bb-c75ad7d0dc6f" containerID="618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b" exitCode=0
Nov 01 09:15:43.871777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.871607    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8" event={"ID":"092a625d-0a4f-4790-95bb-c75ad7d0dc6f","Type":"ContainerDied","Data":"618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b"}
Nov 01 09:15:43.871777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.871628    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"
Nov 01 09:15:43.871777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.871675    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8" event={"ID":"092a625d-0a4f-4790-95bb-c75ad7d0dc6f","Type":"ContainerDied","Data":"dae69169823989350f8e9aa3a9cc4ff37b646a53121bb16b8a7aac6b3a05a08a"}
Nov 01 09:15:43.871777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.871705    4314 scope.go:122] "RemoveContainer" containerID="618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b"
Nov 01 09:15:43.882822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.882805    4314 scope.go:122] "RemoveContainer" containerID="618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b"
Nov 01 09:15:43.883125 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:43.883106    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b\": container with ID starting with 618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b not found: ID does not exist" containerID="618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b"
Nov 01 09:15:43.883199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:43.883136    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b"} err="failed to get container status \"618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b\": rpc error: code = NotFound desc = could not find container \"618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b\": container with ID starting with 618ba154b905e4e1895d4c407b52aa3777c9453dee2c9cc5682755326d02b22b not found: ID does not exist"
Nov 01 09:15:44.027552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:44.027492    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"]
Nov 01 09:15:44.032688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:44.032654    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-6f76f6c75-l7gj8"]
Nov 01 09:15:44.930143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:44.930110    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092a625d-0a4f-4790-95bb-c75ad7d0dc6f" path="/var/lib/kubelet/pods/092a625d-0a4f-4790-95bb-c75ad7d0dc6f/volumes"
Nov 01 09:15:45.139909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.139867    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:45.288005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.287920    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:45.729598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.729494    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:45.887213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.887175    4314 generic.go:356] "Generic (PLEG): container finished" podID="16c92efb-3798-491a-ac10-165626a4f87f" containerID="99e471e65f07de1b92cdabc5e02cdc5360e9763c76bcb39dd783af68855ce64e" exitCode=0
Nov 01 09:15:45.887392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.887249    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" event={"ID":"16c92efb-3798-491a-ac10-165626a4f87f","Type":"ContainerDied","Data":"99e471e65f07de1b92cdabc5e02cdc5360e9763c76bcb39dd783af68855ce64e"}
Nov 01 09:15:45.887714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.887700    4314 scope.go:122] "RemoveContainer" containerID="99e471e65f07de1b92cdabc5e02cdc5360e9763c76bcb39dd783af68855ce64e"
Nov 01 09:15:45.889227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.889195    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:15:45.889308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.889267    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:15:45.889308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:45.889298    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:15:46.720812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:46.720779    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:46.895481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:46.895446    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" event={"ID":"16c92efb-3798-491a-ac10-165626a4f87f","Type":"ContainerStarted","Data":"c690473df16882fad5abcb85a1843e468b49116816fcab8a5c2864fd27bd8fd7"}
Nov 01 09:15:47.196687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:47.196639    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"]
Nov 01 09:15:47.196963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:47.196919    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy" containerID="cri-o://be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed" gracePeriod=30
Nov 01 09:15:47.223094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:47.223071    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:47.286731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:47.286691    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:15:47.287037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:47.287012    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 09:15:47.287147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:47.287080    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 09:15:48.287317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:48.287276    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 09:15:48.287751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:48.287342    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 09:15:48.606323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:48.606230    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:49.249524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:49.249482    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:49.289157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:49.288969    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 09:15:49.289157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:49.289039    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 09:15:49.574366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:49.574251    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:50.286891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:50.286852    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 09:15:50.287083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:50.286909    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 09:15:50.948379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:50.948346    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:51.286600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:51.286561    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 09:15:51.286802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:51.286617    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 09:15:52.291089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.291056    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:15:52.291562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.291541    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:15:52.292314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.292293    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:15:52.457498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.457473    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:15:52.553041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553002    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553068    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553112    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553192    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553228    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553256    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553297    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553329    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553468    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2\") pod \"f536bc1f-0a5f-4109-8296-ad10e2687187\" (UID: \"f536bc1f-0a5f-4109-8296-ad10e2687187\") "
Nov 01 09:15:52.553670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553483    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:15:52.553670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553519    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:15:52.553670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553561    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:15:52.553869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553732    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:15:52.553869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.553809    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:52.554047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.554019    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.554047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.554047    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.554269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.554062    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536bc1f-0a5f-4109-8296-ad10e2687187-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.554269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.554077    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.554269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.554091    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.555434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.555408    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:52.555577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.555454    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:15:52.555577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.555476    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:15:52.555668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.555623    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2" pod "f536bc1f-0a5f-4109-8296-ad10e2687187" (UID: "f536bc1f-0a5f-4109-8296-ad10e2687187"). InnerVolumeSpecName "kube-api-access-rfdq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:52.655285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.655254    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfdq2\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-kube-api-access-rfdq2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.655285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.655281    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.655285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.655292    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.655549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.655303    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536bc1f-0a5f-4109-8296-ad10e2687187-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:52.934041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.933956    4314 generic.go:356] "Generic (PLEG): container finished" podID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerID="be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed" exitCode=0
Nov 01 09:15:52.934041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.934002    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"
Nov 01 09:15:52.934296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.934035    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" event={"ID":"f536bc1f-0a5f-4109-8296-ad10e2687187","Type":"ContainerDied","Data":"be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed"}
Nov 01 09:15:52.934296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.934087    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g" event={"ID":"f536bc1f-0a5f-4109-8296-ad10e2687187","Type":"ContainerDied","Data":"5d992eb7db64395d6a8f5703b3ffcbc64248be1aeaf510c24fc324fe63599ea5"}
Nov 01 09:15:52.934296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.934108    4314 scope.go:122] "RemoveContainer" containerID="be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed"
Nov 01 09:15:52.946550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.946520    4314 scope.go:122] "RemoveContainer" containerID="279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"
Nov 01 09:15:52.956231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.956159    4314 scope.go:122] "RemoveContainer" containerID="be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed"
Nov 01 09:15:52.958721 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:52.958698    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed\": container with ID starting with be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed not found: ID does not exist" containerID="be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed"
Nov 01 09:15:52.958844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.958731    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed"} err="failed to get container status \"be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed\": rpc error: code = NotFound desc = could not find container \"be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed\": container with ID starting with be4293a254d55e25eb73747a686b28beaa7d7bd0087d4446665499e4d97821ed not found: ID does not exist"
Nov 01 09:15:52.958844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.958748    4314 scope.go:122] "RemoveContainer" containerID="279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"
Nov 01 09:15:52.959139 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:15:52.959114    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf\": container with ID starting with 279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf not found: ID does not exist" containerID="279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"
Nov 01 09:15:52.959205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:52.959147    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf"} err="failed to get container status \"279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf\": rpc error: code = NotFound desc = could not find container \"279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf\": container with ID starting with 279ca2d09b20ef00fe64a765f584f7ec427710abd4104ae669dd58b1da9ee6bf not found: ID does not exist"
Nov 01 09:15:53.090188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:53.090139    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"]
Nov 01 09:15:53.135576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:53.135547    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be7cc-openshift-default-7cd76b4d77-xcv7g"]
Nov 01 09:15:53.213132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:53.213040    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:53.402664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:53.402636    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:53.563833 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:53.563797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:54.459815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:54.459779    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:54.608135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:54.608084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:54.929230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:54.929195    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" path="/var/lib/kubelet/pods/f536bc1f-0a5f-4109-8296-ad10e2687187/volumes"
Nov 01 09:15:55.602147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602115    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"]
Nov 01 09:15:55.602878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602862    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ce49ef46-8906-467f-b4d1-6898b4c3cba0" containerName="revoke-keys"
Nov 01 09:15:55.602929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602883    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce49ef46-8906-467f-b4d1-6898b4c3cba0" containerName="revoke-keys"
Nov 01 09:15:55.602929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602906    4314 cpu_manager.go:414] "containerMap: removing container" podUID="092a625d-0a4f-4790-95bb-c75ad7d0dc6f" containerName="payload-pre-processing"
Nov 01 09:15:55.602929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602913    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092a625d-0a4f-4790-95bb-c75ad7d0dc6f" containerName="payload-pre-processing"
Nov 01 09:15:55.603085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602934    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy"
Nov 01 09:15:55.603085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602941    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy"
Nov 01 09:15:55.603085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602954    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy"
Nov 01 09:15:55.603085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.602961    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy"
Nov 01 09:15:55.603288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.603106    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce49ef46-8906-467f-b4d1-6898b4c3cba0" containerName="revoke-keys"
Nov 01 09:15:55.603288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.603117    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="092a625d-0a4f-4790-95bb-c75ad7d0dc6f" containerName="payload-pre-processing"
Nov 01 09:15:55.603288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.603129    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy"
Nov 01 09:15:55.603288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.603138    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f536bc1f-0a5f-4109-8296-ad10e2687187" containerName="istio-proxy"
Nov 01 09:15:55.608491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.608472    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.656891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.656855    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:15:55.658645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.658616    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"]
Nov 01 09:15:55.690308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.690270    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.690480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.690395    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.690480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.690454    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p499g\" (UniqueName: \"kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.700629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.700603    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:15:55.700629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.700611    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:15:55.700629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.700634    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:15:55.791492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.791459    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.791698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.791556    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.791698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.791615    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p499g\" (UniqueName: \"kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.792176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.792153    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.793998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.793971    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.848125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.848090    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p499g\" (UniqueName: \"kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g\") pod \"maas-api-revoke-sub-f4bf36bc-t7nrc\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:55.924202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:55.924117    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:56.119943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.119910    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:56.292944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.292914    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"]
Nov 01 09:15:56.293934 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:56.293909    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723dc0dd_c4c6_45d7_8969_a040b1ea1da4.slice/crio-f5b227c57513e498f7c75af601eb826125b9675c109ef23d4bb0bbfc7cfee74f WatchSource:0}: Error finding container f5b227c57513e498f7c75af601eb826125b9675c109ef23d4bb0bbfc7cfee74f: Status 404 returned error can't find the container with id f5b227c57513e498f7c75af601eb826125b9675c109ef23d4bb0bbfc7cfee74f
Nov 01 09:15:56.375872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.375830    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:56.552384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.552290    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:56.759075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.759039    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:56.779775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.779743    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"]
Nov 01 09:15:56.783845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.783827    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:56.833947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.833870    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0468a5\""
Nov 01 09:15:56.834133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.834099    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0468a5-dockercfg-6rth6\""
Nov 01 09:15:56.837905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.837880    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"]
Nov 01 09:15:56.917838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.917804    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:56.918026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.917870    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9kqq\" (UniqueName: \"kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq\") pod \"payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:56.941387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.941337    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"]
Nov 01 09:15:56.945685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.945665    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:56.960023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.959995    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc" event={"ID":"723dc0dd-c4c6-45d7-8969-a040b1ea1da4","Type":"ContainerStarted","Data":"aae730907fa4c7160a53db17ebbdf6fe9f85d8019b487dcd3dc891432be46fed"}
Nov 01 09:15:56.960161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:56.960037    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc" event={"ID":"723dc0dd-c4c6-45d7-8969-a040b1ea1da4","Type":"ContainerStarted","Data":"f5b227c57513e498f7c75af601eb826125b9675c109ef23d4bb0bbfc7cfee74f"}
Nov 01 09:15:57.000814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.000777    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"]
Nov 01 09:15:57.019419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.019384    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:57.019629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.019599    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwt9j\" (UniqueName: \"kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j\") pod \"payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.019733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.019685    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.019818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.019798    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9kqq\" (UniqueName: \"kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq\") pod \"payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:57.020184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.020166    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:57.076726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.076696    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9kqq\" (UniqueName: \"kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq\") pod \"payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:57.097714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.097646    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:57.120897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.120867    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwt9j\" (UniqueName: \"kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j\") pod \"payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.121037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.120906    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.121874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.121850    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.181120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.181089    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwt9j\" (UniqueName: \"kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j\") pod \"payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.259883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.259852    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.269671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.269638    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:15:57.472133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.472078    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc" podStartSLOduration=2.472060914 podStartE2EDuration="2.472060914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:57.166186814 +0000 UTC m=+2630.360581369" watchObservedRunningTime="2026-11-01 09:15:57.472060914 +0000 UTC m=+2630.666455468"
Nov 01 09:15:57.472732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.472706    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"]
Nov 01 09:15:57.474184 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:57.474160    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f1a00c_619d_4eec_860b_81f599a4c3e2.slice/crio-44b57791aa6c0f98c74e2832cc394234ec813924d911471df5bb1f09482edc59 WatchSource:0}: Error finding container 44b57791aa6c0f98c74e2832cc394234ec813924d911471df5bb1f09482edc59: Status 404 returned error can't find the container with id 44b57791aa6c0f98c74e2832cc394234ec813924d911471df5bb1f09482edc59
Nov 01 09:15:57.627419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.627396    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"]
Nov 01 09:15:57.628463 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:15:57.628437    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b76cc2_8460_4ab3_8e8f_440dfc78929f.slice/crio-a08e622fd49d14e238d1436815adf8f528a89b7b7a133d25c354558d25130e9d WatchSource:0}: Error finding container a08e622fd49d14e238d1436815adf8f528a89b7b7a133d25c354558d25130e9d: Status 404 returned error can't find the container with id a08e622fd49d14e238d1436815adf8f528a89b7b7a133d25c354558d25130e9d
Nov 01 09:15:57.968676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.968641    4314 generic.go:356] "Generic (PLEG): container finished" podID="723dc0dd-c4c6-45d7-8969-a040b1ea1da4" containerID="aae730907fa4c7160a53db17ebbdf6fe9f85d8019b487dcd3dc891432be46fed" exitCode=0
Nov 01 09:15:57.969160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.968728    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc" event={"ID":"723dc0dd-c4c6-45d7-8969-a040b1ea1da4","Type":"ContainerDied","Data":"aae730907fa4c7160a53db17ebbdf6fe9f85d8019b487dcd3dc891432be46fed"}
Nov 01 09:15:57.970397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.970368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh" event={"ID":"65b76cc2-8460-4ab3-8e8f-440dfc78929f","Type":"ContainerStarted","Data":"b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f"}
Nov 01 09:15:57.970573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.970420    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh" event={"ID":"65b76cc2-8460-4ab3-8e8f-440dfc78929f","Type":"ContainerStarted","Data":"a08e622fd49d14e238d1436815adf8f528a89b7b7a133d25c354558d25130e9d"}
Nov 01 09:15:57.970573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.970451    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:15:57.971813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.971789    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf" event={"ID":"24f1a00c-619d-4eec-860b-81f599a4c3e2","Type":"ContainerStarted","Data":"9ae066937dfae2e1a3d0652bc2d7f325d4c56ac30df3773919a8928d5d74ab0c"}
Nov 01 09:15:57.971948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.971827    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf" event={"ID":"24f1a00c-619d-4eec-860b-81f599a4c3e2","Type":"ContainerStarted","Data":"44b57791aa6c0f98c74e2832cc394234ec813924d911471df5bb1f09482edc59"}
Nov 01 09:15:57.971948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:57.971857    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:15:58.186845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:58.186788    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh" podStartSLOduration=2.186775464 podStartE2EDuration="2.186775464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:58.185272464 +0000 UTC m=+2631.379667020" watchObservedRunningTime="2026-11-01 09:15:58.186775464 +0000 UTC m=+2631.381170043"
Nov 01 09:15:58.296224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:58.296173    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf" podStartSLOduration=2.296159023 podStartE2EDuration="2.296159023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:58.292627929 +0000 UTC m=+2631.487022480" watchObservedRunningTime="2026-11-01 09:15:58.296159023 +0000 UTC m=+2631.490553578"
Nov 01 09:15:59.141089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.141063    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:59.257777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.257743    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca\") pod \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") "
Nov 01 09:15:59.257984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.257822    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca\") pod \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") "
Nov 01 09:15:59.258054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.258005    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g\" (UniqueName: \"kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g\") pod \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\" (UID: \"723dc0dd-c4c6-45d7-8969-a040b1ea1da4\") "
Nov 01 09:15:59.258206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.258180    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca" pod "723dc0dd-c4c6-45d7-8969-a040b1ea1da4" (UID: "723dc0dd-c4c6-45d7-8969-a040b1ea1da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:59.258439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.258422    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:59.259984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.259958    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca" pod "723dc0dd-c4c6-45d7-8969-a040b1ea1da4" (UID: "723dc0dd-c4c6-45d7-8969-a040b1ea1da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:59.260230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.260209    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g" pod "723dc0dd-c4c6-45d7-8969-a040b1ea1da4" (UID: "723dc0dd-c4c6-45d7-8969-a040b1ea1da4"). InnerVolumeSpecName "kube-api-access-p499g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:59.359236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.359195    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p499g\" (UniqueName: \"kubernetes.io/projected/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-kube-api-access-p499g\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:59.359236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.359229    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723dc0dd-c4c6-45d7-8969-a040b1ea1da4-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:59.985193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.985162    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"
Nov 01 09:15:59.985379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.985201    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc" event={"ID":"723dc0dd-c4c6-45d7-8969-a040b1ea1da4","Type":"ContainerDied","Data":"f5b227c57513e498f7c75af601eb826125b9675c109ef23d4bb0bbfc7cfee74f"}
Nov 01 09:15:59.985379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:15:59.985249    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b227c57513e498f7c75af601eb826125b9675c109ef23d4bb0bbfc7cfee74f"
Nov 01 09:16:01.336021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:01.335971    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:06.741121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:06.741067    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:08.981684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:08.981656    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:16:08.982138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:08.981723    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:16:09.433328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.433289    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"]
Nov 01 09:16:09.434117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.434097    4314 cpu_manager.go:414] "containerMap: removing container" podUID="723dc0dd-c4c6-45d7-8969-a040b1ea1da4" containerName="revoke-keys"
Nov 01 09:16:09.434189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.434121    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723dc0dd-c4c6-45d7-8969-a040b1ea1da4" containerName="revoke-keys"
Nov 01 09:16:09.434332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.434320    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="723dc0dd-c4c6-45d7-8969-a040b1ea1da4" containerName="revoke-keys"
Nov 01 09:16:09.440002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.439981    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.489242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.489208    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:16:09.489415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.489282    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:16:09.489415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.489370    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:16:09.489415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.489223    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:16:09.489837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.489809    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"]
Nov 01 09:16:09.600430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.600392    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.600800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.600773    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.600923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.600908    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5vk\" (UniqueName: \"kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.702493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.702390    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.702493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.702436    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5vk\" (UniqueName: \"kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.702770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.702571    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.703163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.703136    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.704984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.704956    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:09.759692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:09.759651    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5vk\" (UniqueName: \"kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk\") pod \"maas-api-revoke-sub-2b6d862d-q6xq4\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:10.053596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:10.053564    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:10.431299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:10.431266    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"]
Nov 01 09:16:10.432498 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:10.432471    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8c3674_de68_4eea_a396_b8723f06a920.slice/crio-e3cea2f56d6c49cd2da5ffc1e39ddc3d84f71c233b258ec0d28baae5a19fcdeb WatchSource:0}: Error finding container e3cea2f56d6c49cd2da5ffc1e39ddc3d84f71c233b258ec0d28baae5a19fcdeb: Status 404 returned error can't find the container with id e3cea2f56d6c49cd2da5ffc1e39ddc3d84f71c233b258ec0d28baae5a19fcdeb
Nov 01 09:16:11.059517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:11.059424    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4" event={"ID":"1e8c3674-de68-4eea-a396-b8723f06a920","Type":"ContainerStarted","Data":"0bb47b9ea1b5de09abe69a36fbf0210055a7665b959de849fc35f60a06448145"}
Nov 01 09:16:11.059517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:11.059483    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4" event={"ID":"1e8c3674-de68-4eea-a396-b8723f06a920","Type":"ContainerStarted","Data":"e3cea2f56d6c49cd2da5ffc1e39ddc3d84f71c233b258ec0d28baae5a19fcdeb"}
Nov 01 09:16:11.170978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:11.170919    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4" podStartSLOduration=2.170902866 podStartE2EDuration="2.170902866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:11.167511031 +0000 UTC m=+2644.361905598" watchObservedRunningTime="2026-11-01 09:16:11.170902866 +0000 UTC m=+2644.365297420"
Nov 01 09:16:11.325325 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:16:11.325239    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:16:11.686959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:11.686875    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:12.067914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.067885    4314 generic.go:356] "Generic (PLEG): container finished" podID="1e8c3674-de68-4eea-a396-b8723f06a920" containerID="0bb47b9ea1b5de09abe69a36fbf0210055a7665b959de849fc35f60a06448145" exitCode=0
Nov 01 09:16:12.081503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.067976    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4" event={"ID":"1e8c3674-de68-4eea-a396-b8723f06a920","Type":"ContainerDied","Data":"0bb47b9ea1b5de09abe69a36fbf0210055a7665b959de849fc35f60a06448145"}
Nov 01 09:16:12.554304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.554266    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"]
Nov 01 09:16:12.558182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.558162    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.611384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.611350    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"]
Nov 01 09:16:12.633855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.633805    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.634042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.633967    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mrdb\" (UniqueName: \"kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.634042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.634006    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.735134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.735090    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.735337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.735239    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrdb\" (UniqueName: \"kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.735337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.735281    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.735857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.735834    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.737746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.737718    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.792165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.792130    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrdb\" (UniqueName: \"kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb\") pod \"maas-api-revoke-sub-06908f1d-ws24j\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:12.871396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:12.871315    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:13.075251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.075214    4314 generic.go:356] "Generic (PLEG): container finished" podID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerID="55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa" exitCode=0
Nov 01 09:16:13.075709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.075287    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" event={"ID":"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0","Type":"ContainerDied","Data":"55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"}
Nov 01 09:16:13.075844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.075797    4314 scope.go:122] "RemoveContainer" containerID="55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"
Nov 01 09:16:13.077252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.077217    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:16:13.077355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.077288    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:16:13.077355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.077346    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:16:13.228602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.228572    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:13.238424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.238397    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"]
Nov 01 09:16:13.241851 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:13.240103    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e3e35b_5bc8_43d7_b492_20fe8a2f9115.slice/crio-6ea5e0af51483174b95987940fe427747e300eace71fd05d78b4ed8fc62d41a6 WatchSource:0}: Error finding container 6ea5e0af51483174b95987940fe427747e300eace71fd05d78b4ed8fc62d41a6: Status 404 returned error can't find the container with id 6ea5e0af51483174b95987940fe427747e300eace71fd05d78b4ed8fc62d41a6
Nov 01 09:16:13.242072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.241938    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca\") pod \"1e8c3674-de68-4eea-a396-b8723f06a920\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") "
Nov 01 09:16:13.242072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.242049    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca\") pod \"1e8c3674-de68-4eea-a396-b8723f06a920\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") "
Nov 01 09:16:13.242195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.242178    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk\" (UniqueName: \"kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk\") pod \"1e8c3674-de68-4eea-a396-b8723f06a920\" (UID: \"1e8c3674-de68-4eea-a396-b8723f06a920\") "
Nov 01 09:16:13.242857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.242818    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca" pod "1e8c3674-de68-4eea-a396-b8723f06a920" (UID: "1e8c3674-de68-4eea-a396-b8723f06a920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:13.244976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.244947    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca" pod "1e8c3674-de68-4eea-a396-b8723f06a920" (UID: "1e8c3674-de68-4eea-a396-b8723f06a920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:13.245063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.244967    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk" pod "1e8c3674-de68-4eea-a396-b8723f06a920" (UID: "1e8c3674-de68-4eea-a396-b8723f06a920"). InnerVolumeSpecName "kube-api-access-hx5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:13.343494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.343460    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:13.343494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.343494    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5vk\" (UniqueName: \"kubernetes.io/projected/1e8c3674-de68-4eea-a396-b8723f06a920-kube-api-access-hx5vk\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:13.343781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:13.343508    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c3674-de68-4eea-a396-b8723f06a920-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:14.107306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.107275    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"
Nov 01 09:16:14.107715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.107318    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4" event={"ID":"1e8c3674-de68-4eea-a396-b8723f06a920","Type":"ContainerDied","Data":"e3cea2f56d6c49cd2da5ffc1e39ddc3d84f71c233b258ec0d28baae5a19fcdeb"}
Nov 01 09:16:14.107715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.107394    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3cea2f56d6c49cd2da5ffc1e39ddc3d84f71c233b258ec0d28baae5a19fcdeb"
Nov 01 09:16:14.109567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.109527    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" event={"ID":"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0","Type":"ContainerStarted","Data":"d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae"}
Nov 01 09:16:14.111162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.111142    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j" event={"ID":"09e3e35b-5bc8-43d7-b492-20fe8a2f9115","Type":"ContainerStarted","Data":"13d8b812f4627cffd6b0f2a0424aa5398331933785ee295b80efbcec71122b05"}
Nov 01 09:16:14.111255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.111177    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j" event={"ID":"09e3e35b-5bc8-43d7-b492-20fe8a2f9115","Type":"ContainerStarted","Data":"6ea5e0af51483174b95987940fe427747e300eace71fd05d78b4ed8fc62d41a6"}
Nov 01 09:16:14.341636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.341587    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j" podStartSLOduration=2.341572234 podStartE2EDuration="2.341572234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:14.339293543 +0000 UTC m=+2647.533688099" watchObservedRunningTime="2026-11-01 09:16:14.341572234 +0000 UTC m=+2647.535966793"
Nov 01 09:16:14.998973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:14.998937    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:15.103032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:15.102989    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:16:15.103825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:15.103798    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 09:16:15.103950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:15.103848    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 09:16:15.119021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:15.118992    4314 generic.go:356] "Generic (PLEG): container finished" podID="09e3e35b-5bc8-43d7-b492-20fe8a2f9115" containerID="13d8b812f4627cffd6b0f2a0424aa5398331933785ee295b80efbcec71122b05" exitCode=0
Nov 01 09:16:15.119470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:15.119084    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j" event={"ID":"09e3e35b-5bc8-43d7-b492-20fe8a2f9115","Type":"ContainerDied","Data":"13d8b812f4627cffd6b0f2a0424aa5398331933785ee295b80efbcec71122b05"}
Nov 01 09:16:16.103593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.103515    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 09:16:16.103873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.103609    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 09:16:16.313144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.313113    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:16.402694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.402604    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca\") pod \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") "
Nov 01 09:16:16.402694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.402645    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca\") pod \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") "
Nov 01 09:16:16.402958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.402713    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb\" (UniqueName: \"kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb\") pod \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\" (UID: \"09e3e35b-5bc8-43d7-b492-20fe8a2f9115\") "
Nov 01 09:16:16.403089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.403064    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca" pod "09e3e35b-5bc8-43d7-b492-20fe8a2f9115" (UID: "09e3e35b-5bc8-43d7-b492-20fe8a2f9115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:16.404799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.404775    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca" pod "09e3e35b-5bc8-43d7-b492-20fe8a2f9115" (UID: "09e3e35b-5bc8-43d7-b492-20fe8a2f9115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:16.404883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.404859    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb" pod "09e3e35b-5bc8-43d7-b492-20fe8a2f9115" (UID: "09e3e35b-5bc8-43d7-b492-20fe8a2f9115"). InnerVolumeSpecName "kube-api-access-6mrdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:16.504094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.504048    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mrdb\" (UniqueName: \"kubernetes.io/projected/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-kube-api-access-6mrdb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:16.504094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.504092    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:16.504310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.504105    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e3e35b-5bc8-43d7-b492-20fe8a2f9115-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:16.585379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:16.585191    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:17.103245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:17.103204    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 09:16:17.103432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:17.103258    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 09:16:17.133909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:17.133873    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j" event={"ID":"09e3e35b-5bc8-43d7-b492-20fe8a2f9115","Type":"ContainerDied","Data":"6ea5e0af51483174b95987940fe427747e300eace71fd05d78b4ed8fc62d41a6"}
Nov 01 09:16:17.133909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:17.133898    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"
Nov 01 09:16:17.134139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:17.133934    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea5e0af51483174b95987940fe427747e300eace71fd05d78b4ed8fc62d41a6"
Nov 01 09:16:18.103877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:18.103840    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 09:16:18.104400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:18.103907    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 09:16:19.102976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:19.102883    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 09:16:19.102976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:19.102949    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 09:16:20.107696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.107664    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:16:20.108124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.107955    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:16:20.108733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.108710    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:16:20.205417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.205377    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"]
Nov 01 09:16:20.205951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.205934    4314 cpu_manager.go:414] "containerMap: removing container" podUID="09e3e35b-5bc8-43d7-b492-20fe8a2f9115" containerName="revoke-keys"
Nov 01 09:16:20.206004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.205952    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e3e35b-5bc8-43d7-b492-20fe8a2f9115" containerName="revoke-keys"
Nov 01 09:16:20.206004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.205984    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1e8c3674-de68-4eea-a396-b8723f06a920" containerName="revoke-keys"
Nov 01 09:16:20.206004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.205991    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8c3674-de68-4eea-a396-b8723f06a920" containerName="revoke-keys"
Nov 01 09:16:20.206113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.206084    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e3e35b-5bc8-43d7-b492-20fe8a2f9115" containerName="revoke-keys"
Nov 01 09:16:20.206113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.206097    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8c3674-de68-4eea-a396-b8723f06a920" containerName="revoke-keys"
Nov 01 09:16:20.209417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.209401    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.239807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.239782    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"]
Nov 01 09:16:20.244837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.244802    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqzg\" (UniqueName: \"kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.245046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.245021    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.245276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.245246    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.260249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.260218    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:16:20.260249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.260241    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:16:20.260482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.260237    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:16:20.260482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.260286    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:16:20.346273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.346228    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.346480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.346346    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.346480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.346417    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqzg\" (UniqueName: \"kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.347113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.347091    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.348731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.348712    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.403876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.403790    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqzg\" (UniqueName: \"kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg\") pod \"maas-api-revoke-sub-e444f639-25wb6\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.523275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.523233    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:20.915697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:20.915662    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"]
Nov 01 09:16:20.916716 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:20.916682    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b5b977_2e11_485c_8d15_a83f509df500.slice/crio-1d7e62bd340987e28b968cf03583b0358ee0b8ee2f17efe05674fbdff04209f8 WatchSource:0}: Error finding container 1d7e62bd340987e28b968cf03583b0358ee0b8ee2f17efe05674fbdff04209f8: Status 404 returned error can't find the container with id 1d7e62bd340987e28b968cf03583b0358ee0b8ee2f17efe05674fbdff04209f8
Nov 01 09:16:21.160372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:21.160334    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6" event={"ID":"72b5b977-2e11-485c-8d15-a83f509df500","Type":"ContainerStarted","Data":"1d7e62bd340987e28b968cf03583b0358ee0b8ee2f17efe05674fbdff04209f8"}
Nov 01 09:16:21.485728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:21.485694    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:22.168108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:22.168019    4314 generic.go:356] "Generic (PLEG): container finished" podID="72b5b977-2e11-485c-8d15-a83f509df500" containerID="c6e1c9a990e22d60060c4349e87027ef1acbd90d9083f90bb143609fa20ad197" exitCode=0
Nov 01 09:16:22.168511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:22.168105    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6" event={"ID":"72b5b977-2e11-485c-8d15-a83f509df500","Type":"ContainerDied","Data":"c6e1c9a990e22d60060c4349e87027ef1acbd90d9083f90bb143609fa20ad197"}
Nov 01 09:16:23.326861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.326834    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:23.388176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.388141    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca\") pod \"72b5b977-2e11-485c-8d15-a83f509df500\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") "
Nov 01 09:16:23.388402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.388329    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg\" (UniqueName: \"kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg\") pod \"72b5b977-2e11-485c-8d15-a83f509df500\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") "
Nov 01 09:16:23.388402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.388396    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca\") pod \"72b5b977-2e11-485c-8d15-a83f509df500\" (UID: \"72b5b977-2e11-485c-8d15-a83f509df500\") "
Nov 01 09:16:23.388613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.388586    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca" pod "72b5b977-2e11-485c-8d15-a83f509df500" (UID: "72b5b977-2e11-485c-8d15-a83f509df500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:23.388893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.388873    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:23.390498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.390479    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca" pod "72b5b977-2e11-485c-8d15-a83f509df500" (UID: "72b5b977-2e11-485c-8d15-a83f509df500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:23.390620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.390605    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg" pod "72b5b977-2e11-485c-8d15-a83f509df500" (UID: "72b5b977-2e11-485c-8d15-a83f509df500"). InnerVolumeSpecName "kube-api-access-kwqzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:23.459189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.459108    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"]
Nov 01 09:16:23.459657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.459643    4314 cpu_manager.go:414] "containerMap: removing container" podUID="72b5b977-2e11-485c-8d15-a83f509df500" containerName="revoke-keys"
Nov 01 09:16:23.459705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.459658    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b5b977-2e11-485c-8d15-a83f509df500" containerName="revoke-keys"
Nov 01 09:16:23.459758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.459748    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b5b977-2e11-485c-8d15-a83f509df500" containerName="revoke-keys"
Nov 01 09:16:23.463173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.463151    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.490081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.490049    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwqzg\" (UniqueName: \"kubernetes.io/projected/72b5b977-2e11-485c-8d15-a83f509df500-kube-api-access-kwqzg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:23.490081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.490081    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b5b977-2e11-485c-8d15-a83f509df500-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:23.542841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.542809    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"]
Nov 01 09:16:23.591126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.591087    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.591126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.591129    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.591363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.591222    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chkrr\" (UniqueName: \"kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.692638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.692605    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.692638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.692647    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.692917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.692696    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chkrr\" (UniqueName: \"kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.693262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.693240    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.695115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.695091    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.750112 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.750075    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chkrr\" (UniqueName: \"kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr\") pod \"maas-api-revoke-sub-5333af69-ltv8z\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:23.776767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:23.776736    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:24.146618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:24.146585    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"]
Nov 01 09:16:24.148063 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:24.148034    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25209b9a_10d8_4903_9202_1ec041b5dab3.slice/crio-817512bb7d3dd60110e9e346e82169d3f1d66e844bdeca96687f239856edca88 WatchSource:0}: Error finding container 817512bb7d3dd60110e9e346e82169d3f1d66e844bdeca96687f239856edca88: Status 404 returned error can't find the container with id 817512bb7d3dd60110e9e346e82169d3f1d66e844bdeca96687f239856edca88
Nov 01 09:16:24.181433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:24.181392    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z" event={"ID":"25209b9a-10d8-4903-9202-1ec041b5dab3","Type":"ContainerStarted","Data":"817512bb7d3dd60110e9e346e82169d3f1d66e844bdeca96687f239856edca88"}
Nov 01 09:16:24.183024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:24.182996    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6" event={"ID":"72b5b977-2e11-485c-8d15-a83f509df500","Type":"ContainerDied","Data":"1d7e62bd340987e28b968cf03583b0358ee0b8ee2f17efe05674fbdff04209f8"}
Nov 01 09:16:24.183167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:24.183037    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"
Nov 01 09:16:24.183167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:24.183041    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d7e62bd340987e28b968cf03583b0358ee0b8ee2f17efe05674fbdff04209f8"
Nov 01 09:16:24.423228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:24.423143    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:25.190275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:25.190242    4314 generic.go:356] "Generic (PLEG): container finished" podID="25209b9a-10d8-4903-9202-1ec041b5dab3" containerID="f337150745fccac2996feebe2f6dfcf40da938a443239aafae990b929921decb" exitCode=0
Nov 01 09:16:25.190482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:25.190323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z" event={"ID":"25209b9a-10d8-4903-9202-1ec041b5dab3","Type":"ContainerDied","Data":"f337150745fccac2996feebe2f6dfcf40da938a443239aafae990b929921decb"}
Nov 01 09:16:26.345168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.345138    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:26.424205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.424173    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca\") pod \"25209b9a-10d8-4903-9202-1ec041b5dab3\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") "
Nov 01 09:16:26.424385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.424262    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca\") pod \"25209b9a-10d8-4903-9202-1ec041b5dab3\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") "
Nov 01 09:16:26.424435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.424408    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr\" (UniqueName: \"kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr\") pod \"25209b9a-10d8-4903-9202-1ec041b5dab3\" (UID: \"25209b9a-10d8-4903-9202-1ec041b5dab3\") "
Nov 01 09:16:26.424597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.424576    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca" pod "25209b9a-10d8-4903-9202-1ec041b5dab3" (UID: "25209b9a-10d8-4903-9202-1ec041b5dab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:26.424835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.424814    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:26.426303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.426277    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca" pod "25209b9a-10d8-4903-9202-1ec041b5dab3" (UID: "25209b9a-10d8-4903-9202-1ec041b5dab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:26.426558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.426511    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr" pod "25209b9a-10d8-4903-9202-1ec041b5dab3" (UID: "25209b9a-10d8-4903-9202-1ec041b5dab3"). InnerVolumeSpecName "kube-api-access-chkrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:26.502653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.502619    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:26.526108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.526073    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25209b9a-10d8-4903-9202-1ec041b5dab3-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:26.526108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:26.526102    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chkrr\" (UniqueName: \"kubernetes.io/projected/25209b9a-10d8-4903-9202-1ec041b5dab3-kube-api-access-chkrr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:27.203746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:27.203663    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"
Nov 01 09:16:27.203904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:27.203665    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z" event={"ID":"25209b9a-10d8-4903-9202-1ec041b5dab3","Type":"ContainerDied","Data":"817512bb7d3dd60110e9e346e82169d3f1d66e844bdeca96687f239856edca88"}
Nov 01 09:16:27.203904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:27.203802    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817512bb7d3dd60110e9e346e82169d3f1d66e844bdeca96687f239856edca88"
Nov 01 09:16:28.680449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:28.680415    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:31.377077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:31.377031    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:34.869240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:34.869206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:36.286756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.286719    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"]
Nov 01 09:16:36.287246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.287232    4314 cpu_manager.go:414] "containerMap: removing container" podUID="25209b9a-10d8-4903-9202-1ec041b5dab3" containerName="revoke-keys"
Nov 01 09:16:36.287295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.287247    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25209b9a-10d8-4903-9202-1ec041b5dab3" containerName="revoke-keys"
Nov 01 09:16:36.287372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.287363    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="25209b9a-10d8-4903-9202-1ec041b5dab3" containerName="revoke-keys"
Nov 01 09:16:36.290916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.290897    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.316965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.316934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:36.339350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.339322    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:16:36.339519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.339355    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:16:36.339519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.339428    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:16:36.385635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.385602    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:16:36.389113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.389082    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"]
Nov 01 09:16:36.443738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.443702    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gvm\" (UniqueName: \"kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.443932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.443815    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.443932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.443860    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.545124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.545020    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.545312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.545218    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gvm\" (UniqueName: \"kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.545312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.545280    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.545946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.545922    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.547464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.547444    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.602650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.602616    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gvm\" (UniqueName: \"kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm\") pod \"maas-api-revoke-sub-7fe6e194-9z58d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.604019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.604003    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:36.947396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.947318    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"]
Nov 01 09:16:36.953684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.953643    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:36.975204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:36.975165    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"]
Nov 01 09:16:37.007955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.007920    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-54e01b-openshift-default-dockercfg-gln2n\""
Nov 01 09:16:37.010331 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:37.010280    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdee6922_4d40_447c_a174_ca9281b0304d.slice/crio-3a449f57f32dcfc89e61b030ba123f795ec53fc084f8f7f5e6240a21dd432cf8 WatchSource:0}: Error finding container 3a449f57f32dcfc89e61b030ba123f795ec53fc084f8f7f5e6240a21dd432cf8: Status 404 returned error can't find the container with id 3a449f57f32dcfc89e61b030ba123f795ec53fc084f8f7f5e6240a21dd432cf8
Nov 01 09:16:37.023072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.023034    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"]
Nov 01 09:16:37.056906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.056867    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.056970    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.056999    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.057115    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.057148    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.057180    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.057319    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.057360    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdff\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.057461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.057449    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.158453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.158417    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.158453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.158457    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.158741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.158482    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.158741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.158614    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdff\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.158859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.158830    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.158918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.158901    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159020    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159049    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159086    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159117    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159131    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159191    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159266    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.159392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.159332    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.161367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.161342    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.161629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.161613    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.258354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.258311    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.258587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.258498    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdff\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.269547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.269503    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:37.274240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.274204    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d" event={"ID":"fdee6922-4d40-447c-a174-ca9281b0304d","Type":"ContainerStarted","Data":"3a449f57f32dcfc89e61b030ba123f795ec53fc084f8f7f5e6240a21dd432cf8"}
Nov 01 09:16:37.512385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.512292    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:37.689241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.689204    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"]
Nov 01 09:16:37.690257 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:37.690232    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8196af88_ac13_483a_8911_6a9bff599b3f.slice/crio-2e10602e1fd4e38c921f8f2e0b17da29c4c3899ffb555357e137954da7dcd418 WatchSource:0}: Error finding container 2e10602e1fd4e38c921f8f2e0b17da29c4c3899ffb555357e137954da7dcd418: Status 404 returned error can't find the container with id 2e10602e1fd4e38c921f8f2e0b17da29c4c3899ffb555357e137954da7dcd418
Nov 01 09:16:37.693475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.693436    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:16:37.693586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.693509    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:16:37.693586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:37.693552    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:16:38.287972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:38.287941    4314 generic.go:356] "Generic (PLEG): container finished" podID="fdee6922-4d40-447c-a174-ca9281b0304d" containerID="655124e121d2c24269bc596805b2aa136d18ca12f0715ac9b2956575b30f71d8" exitCode=0
Nov 01 09:16:38.288185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:38.288038    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d" event={"ID":"fdee6922-4d40-447c-a174-ca9281b0304d","Type":"ContainerDied","Data":"655124e121d2c24269bc596805b2aa136d18ca12f0715ac9b2956575b30f71d8"}
Nov 01 09:16:38.289783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:38.289756    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" event={"ID":"8196af88-ac13-483a-8911-6a9bff599b3f","Type":"ContainerStarted","Data":"33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934"}
Nov 01 09:16:38.289931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:38.289812    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" event={"ID":"8196af88-ac13-483a-8911-6a9bff599b3f","Type":"ContainerStarted","Data":"2e10602e1fd4e38c921f8f2e0b17da29c4c3899ffb555357e137954da7dcd418"}
Nov 01 09:16:38.511465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:38.511417    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podStartSLOduration=2.511401743 podStartE2EDuration="2.511401743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:38.508495245 +0000 UTC m=+2671.702889791" watchObservedRunningTime="2026-11-01 09:16:38.511401743 +0000 UTC m=+2671.705796297"
Nov 01 09:16:39.277039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.276954    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:39.282916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.282883    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 09:16:39.283093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.282944    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 09:16:39.440130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.440096    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"]
Nov 01 09:16:39.444358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.444340    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.454928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.454903    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:39.492509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.492472    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca\") pod \"fdee6922-4d40-447c-a174-ca9281b0304d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") "
Nov 01 09:16:39.492719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.492595    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm\" (UniqueName: \"kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm\") pod \"fdee6922-4d40-447c-a174-ca9281b0304d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") "
Nov 01 09:16:39.492719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.492624    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca\") pod \"fdee6922-4d40-447c-a174-ca9281b0304d\" (UID: \"fdee6922-4d40-447c-a174-ca9281b0304d\") "
Nov 01 09:16:39.492837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.492774    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.492837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.492812    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca" pod "fdee6922-4d40-447c-a174-ca9281b0304d" (UID: "fdee6922-4d40-447c-a174-ca9281b0304d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:39.493014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.492993    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rshq\" (UniqueName: \"kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.493095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.493032    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.493143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.493101    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:39.495120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.495094    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca" pod "fdee6922-4d40-447c-a174-ca9281b0304d" (UID: "fdee6922-4d40-447c-a174-ca9281b0304d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:39.495258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.495215    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm" pod "fdee6922-4d40-447c-a174-ca9281b0304d" (UID: "fdee6922-4d40-447c-a174-ca9281b0304d"). InnerVolumeSpecName "kube-api-access-x2gvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:39.532914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.532812    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"]
Nov 01 09:16:39.594670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.594635    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rshq\" (UniqueName: \"kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.594847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.594694    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.594847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.594738    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.594949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.594918    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gvm\" (UniqueName: \"kubernetes.io/projected/fdee6922-4d40-447c-a174-ca9281b0304d-kube-api-access-x2gvm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:39.594949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.594937    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdee6922-4d40-447c-a174-ca9281b0304d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:39.595508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.595483    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.597577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.597545    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.652465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.652434    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rshq\" (UniqueName: \"kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq\") pod \"maas-api-revoke-sub-c672939b-zk89l\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:39.789618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:39.789508    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:40.184622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.184584    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"]
Nov 01 09:16:40.186376 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:40.186349    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb71789_d205_46a9_b636_1df2dba34a9e.slice/crio-5a0d5abfa9954eaeb30046d1be4590cfdcfede718b13276780e680bd4f0cbe0a WatchSource:0}: Error finding container 5a0d5abfa9954eaeb30046d1be4590cfdcfede718b13276780e680bd4f0cbe0a: Status 404 returned error can't find the container with id 5a0d5abfa9954eaeb30046d1be4590cfdcfede718b13276780e680bd4f0cbe0a
Nov 01 09:16:40.277572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.271012    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 09:16:40.277572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.271079    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 09:16:40.305207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.305171    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l" event={"ID":"cdb71789-d205-46a9-b636-1df2dba34a9e","Type":"ContainerStarted","Data":"5a0d5abfa9954eaeb30046d1be4590cfdcfede718b13276780e680bd4f0cbe0a"}
Nov 01 09:16:40.306503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.306482    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"
Nov 01 09:16:40.306640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.306479    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d" event={"ID":"fdee6922-4d40-447c-a174-ca9281b0304d","Type":"ContainerDied","Data":"3a449f57f32dcfc89e61b030ba123f795ec53fc084f8f7f5e6240a21dd432cf8"}
Nov 01 09:16:40.306640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:40.306609    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a449f57f32dcfc89e61b030ba123f795ec53fc084f8f7f5e6240a21dd432cf8"
Nov 01 09:16:41.020005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:41.019969    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:41.279617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:41.278990    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 09:16:41.279617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:41.279057    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 09:16:41.314622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:41.314574    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l" event={"ID":"cdb71789-d205-46a9-b636-1df2dba34a9e","Type":"ContainerStarted","Data":"fffa5f92190761db7222c110f3f2864acec896e2de0e7a086a8ee645d4c948df"}
Nov 01 09:16:41.325660 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:16:41.325622    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:16:41.429079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:41.429026    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l" podStartSLOduration=2.429010449 podStartE2EDuration="2.429010449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:41.427365451 +0000 UTC m=+2674.621760007" watchObservedRunningTime="2026-11-01 09:16:41.429010449 +0000 UTC m=+2674.623405003"
Nov 01 09:16:41.582618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:41.579060    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:42.278558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:42.277987    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 09:16:42.278558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:42.278054    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 09:16:42.322855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:42.322819    4314 generic.go:356] "Generic (PLEG): container finished" podID="cdb71789-d205-46a9-b636-1df2dba34a9e" containerID="fffa5f92190761db7222c110f3f2864acec896e2de0e7a086a8ee645d4c948df" exitCode=0
Nov 01 09:16:42.323256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:42.322899    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l" event={"ID":"cdb71789-d205-46a9-b636-1df2dba34a9e","Type":"ContainerDied","Data":"fffa5f92190761db7222c110f3f2864acec896e2de0e7a086a8ee645d4c948df"}
Nov 01 09:16:43.278556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.269947    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 09:16:43.278556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.270010    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 09:16:43.512182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.512152    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:43.549579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.549433    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca\") pod \"cdb71789-d205-46a9-b636-1df2dba34a9e\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") "
Nov 01 09:16:43.549579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.549491    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca\") pod \"cdb71789-d205-46a9-b636-1df2dba34a9e\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") "
Nov 01 09:16:43.549579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.549571    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq\" (UniqueName: \"kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq\") pod \"cdb71789-d205-46a9-b636-1df2dba34a9e\" (UID: \"cdb71789-d205-46a9-b636-1df2dba34a9e\") "
Nov 01 09:16:43.549918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.549893    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca" pod "cdb71789-d205-46a9-b636-1df2dba34a9e" (UID: "cdb71789-d205-46a9-b636-1df2dba34a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:43.550062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.550047    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:43.551732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.551702    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca" pod "cdb71789-d205-46a9-b636-1df2dba34a9e" (UID: "cdb71789-d205-46a9-b636-1df2dba34a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:43.551849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.551733    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq" pod "cdb71789-d205-46a9-b636-1df2dba34a9e" (UID: "cdb71789-d205-46a9-b636-1df2dba34a9e"). InnerVolumeSpecName "kube-api-access-4rshq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:43.651562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.651484    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rshq\" (UniqueName: \"kubernetes.io/projected/cdb71789-d205-46a9-b636-1df2dba34a9e-kube-api-access-4rshq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:43.651562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:43.651566    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb71789-d205-46a9-b636-1df2dba34a9e-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:44.163436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.163399    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"]
Nov 01 09:16:44.270906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.270870    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 09:16:44.270906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.270942    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 09:16:44.348974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.342029    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"
Nov 01 09:16:44.348974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.342028    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l" event={"ID":"cdb71789-d205-46a9-b636-1df2dba34a9e","Type":"ContainerDied","Data":"5a0d5abfa9954eaeb30046d1be4590cfdcfede718b13276780e680bd4f0cbe0a"}
Nov 01 09:16:44.348974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.343301    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a0d5abfa9954eaeb30046d1be4590cfdcfede718b13276780e680bd4f0cbe0a"
Nov 01 09:16:44.406852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.406808    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:44.550269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:44.550233    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:45.843809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:45.843772    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:45.844270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:45.844185    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:45.844939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:45.844921    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:16:46.261324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:46.261283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:46.420175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:46.420139    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:46.638711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:46.638626    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:49.164417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.164300    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm" podUID="f1c28f4b-375a-48c4-8e21-59d268ec866a" containerName="payload-pre-processing" containerID="cri-o://264689f5c6c96f7042eea6c34d5357abef8ce8c53e35b99368585c53c3ae2e65" gracePeriod=25
Nov 01 09:16:49.344659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.344113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:49.380416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.380381    4314 generic.go:356] "Generic (PLEG): container finished" podID="f1c28f4b-375a-48c4-8e21-59d268ec866a" containerID="264689f5c6c96f7042eea6c34d5357abef8ce8c53e35b99368585c53c3ae2e65" exitCode=0
Nov 01 09:16:49.380613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.380433    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm" event={"ID":"f1c28f4b-375a-48c4-8e21-59d268ec866a","Type":"ContainerDied","Data":"264689f5c6c96f7042eea6c34d5357abef8ce8c53e35b99368585c53c3ae2e65"}
Nov 01 09:16:49.428025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.427963    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:16:49.535273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.535243    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj\" (UniqueName: \"kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj\") pod \"f1c28f4b-375a-48c4-8e21-59d268ec866a\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") "
Nov 01 09:16:49.535481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.535302    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume\") pod \"f1c28f4b-375a-48c4-8e21-59d268ec866a\" (UID: \"f1c28f4b-375a-48c4-8e21-59d268ec866a\") "
Nov 01 09:16:49.535753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.535722    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume" pod "f1c28f4b-375a-48c4-8e21-59d268ec866a" (UID: "f1c28f4b-375a-48c4-8e21-59d268ec866a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:49.537973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.537942    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj" pod "f1c28f4b-375a-48c4-8e21-59d268ec866a" (UID: "f1c28f4b-375a-48c4-8e21-59d268ec866a"). InnerVolumeSpecName "kube-api-access-grzmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:49.637239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.637208    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grzmj\" (UniqueName: \"kubernetes.io/projected/f1c28f4b-375a-48c4-8e21-59d268ec866a-kube-api-access-grzmj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:49.637239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:49.637236    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c28f4b-375a-48c4-8e21-59d268ec866a-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:50.387839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:50.387805    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"
Nov 01 09:16:50.387839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:50.387823    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm" event={"ID":"f1c28f4b-375a-48c4-8e21-59d268ec866a","Type":"ContainerDied","Data":"b252d7eddca5c551627c57c479905dae95272ffe97e3b817db8285e7c97a5f68"}
Nov 01 09:16:50.388497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:50.387896    4314 scope.go:122] "RemoveContainer" containerID="264689f5c6c96f7042eea6c34d5357abef8ce8c53e35b99368585c53c3ae2e65"
Nov 01 09:16:50.542285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:50.542247    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"]
Nov 01 09:16:50.588405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:50.588373    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-28efa1-94c9b99d7-rx6jm"]
Nov 01 09:16:50.929466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:50.929436    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c28f4b-375a-48c4-8e21-59d268ec866a" path="/var/lib/kubelet/pods/f1c28f4b-375a-48c4-8e21-59d268ec866a/volumes"
Nov 01 09:16:51.336392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:51.336361    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:55.564187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:55.564145    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:56.881352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881317    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"]
Nov 01 09:16:56.881868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881852    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cdb71789-d205-46a9-b636-1df2dba34a9e" containerName="revoke-keys"
Nov 01 09:16:56.881914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881869    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb71789-d205-46a9-b636-1df2dba34a9e" containerName="revoke-keys"
Nov 01 09:16:56.881914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881883    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fdee6922-4d40-447c-a174-ca9281b0304d" containerName="revoke-keys"
Nov 01 09:16:56.881914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881889    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdee6922-4d40-447c-a174-ca9281b0304d" containerName="revoke-keys"
Nov 01 09:16:56.882016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881926    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f1c28f4b-375a-48c4-8e21-59d268ec866a" containerName="payload-pre-processing"
Nov 01 09:16:56.882016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.881932    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c28f4b-375a-48c4-8e21-59d268ec866a" containerName="payload-pre-processing"
Nov 01 09:16:56.882016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.882004    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb71789-d205-46a9-b636-1df2dba34a9e" containerName="revoke-keys"
Nov 01 09:16:56.882016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.882012    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdee6922-4d40-447c-a174-ca9281b0304d" containerName="revoke-keys"
Nov 01 09:16:56.882142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.882026    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c28f4b-375a-48c4-8e21-59d268ec866a" containerName="payload-pre-processing"
Nov 01 09:16:56.886699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.886681    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:56.935372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.935346    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:16:56.981440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.981407    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:16:56.981605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.981450    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:16:56.981605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.981548    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:16:56.987339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:56.987317    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"]
Nov 01 09:16:57.031942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.031903    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.032133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.032092    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpwr\" (UniqueName: \"kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.032133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.032119    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.133710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.133626    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.133864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.133728    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwr\" (UniqueName: \"kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.133864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.133750    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.134370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.134346    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.136033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.136006    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.191120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.191084    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpwr\" (UniqueName: \"kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr\") pod \"maas-api-revoke-sub-06218628-fxvzp\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.200879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.200857    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:16:57.479633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.479486    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:16:57.568451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:57.568421    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"]
Nov 01 09:16:57.569547 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:16:57.569509    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7c4d40_b881_45d2_84c0_e16becfca818.slice/crio-cf3915a9b4253a4b3f0fc99aac4f2076783e9fcb90958523a1f6f2c4a6af8bdc WatchSource:0}: Error finding container cf3915a9b4253a4b3f0fc99aac4f2076783e9fcb90958523a1f6f2c4a6af8bdc: Status 404 returned error can't find the container with id cf3915a9b4253a4b3f0fc99aac4f2076783e9fcb90958523a1f6f2c4a6af8bdc
Nov 01 09:16:58.452362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:58.452323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp" event={"ID":"7a7c4d40-b881-45d2-84c0-e16becfca818","Type":"ContainerStarted","Data":"d0240e8dba5ddb062bdc1a6498cd749f5d93c35a0c41bb5f51ddeabe9f7d5c19"}
Nov 01 09:16:58.452929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:58.452384    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp" event={"ID":"7a7c4d40-b881-45d2-84c0-e16becfca818","Type":"ContainerStarted","Data":"cf3915a9b4253a4b3f0fc99aac4f2076783e9fcb90958523a1f6f2c4a6af8bdc"}
Nov 01 09:16:58.560885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:58.560828    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp" podStartSLOduration=2.560812302 podStartE2EDuration="2.560812302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:58.559828539 +0000 UTC m=+2691.754223103" watchObservedRunningTime="2026-11-01 09:16:58.560812302 +0000 UTC m=+2691.755206858"
Nov 01 09:16:59.461911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:59.461878    4314 generic.go:356] "Generic (PLEG): container finished" podID="7a7c4d40-b881-45d2-84c0-e16becfca818" containerID="d0240e8dba5ddb062bdc1a6498cd749f5d93c35a0c41bb5f51ddeabe9f7d5c19" exitCode=0
Nov 01 09:16:59.462338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:16:59.461970    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp" event={"ID":"7a7c4d40-b881-45d2-84c0-e16becfca818","Type":"ContainerDied","Data":"d0240e8dba5ddb062bdc1a6498cd749f5d93c35a0c41bb5f51ddeabe9f7d5c19"}
Nov 01 09:17:00.608853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.608827    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:17:00.672556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.672500    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca\") pod \"7a7c4d40-b881-45d2-84c0-e16becfca818\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") "
Nov 01 09:17:00.672746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.672597    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr\" (UniqueName: \"kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr\") pod \"7a7c4d40-b881-45d2-84c0-e16becfca818\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") "
Nov 01 09:17:00.672746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.672634    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca\") pod \"7a7c4d40-b881-45d2-84c0-e16becfca818\" (UID: \"7a7c4d40-b881-45d2-84c0-e16becfca818\") "
Nov 01 09:17:00.672935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.672909    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca" pod "7a7c4d40-b881-45d2-84c0-e16becfca818" (UID: "7a7c4d40-b881-45d2-84c0-e16becfca818"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:00.673054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.673037    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:00.674818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.674787    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr" pod "7a7c4d40-b881-45d2-84c0-e16becfca818" (UID: "7a7c4d40-b881-45d2-84c0-e16becfca818"). InnerVolumeSpecName "kube-api-access-qrpwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:00.674930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.674830    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca" pod "7a7c4d40-b881-45d2-84c0-e16becfca818" (UID: "7a7c4d40-b881-45d2-84c0-e16becfca818"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:17:00.774182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.774146    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7c4d40-b881-45d2-84c0-e16becfca818-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:00.774182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:00.774176    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpwr\" (UniqueName: \"kubernetes.io/projected/7a7c4d40-b881-45d2-84c0-e16becfca818-kube-api-access-qrpwr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:01.475910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:01.475882    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"
Nov 01 09:17:01.476070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:01.475922    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp" event={"ID":"7a7c4d40-b881-45d2-84c0-e16becfca818","Type":"ContainerDied","Data":"cf3915a9b4253a4b3f0fc99aac4f2076783e9fcb90958523a1f6f2c4a6af8bdc"}
Nov 01 09:17:01.476070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:01.475972    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf3915a9b4253a4b3f0fc99aac4f2076783e9fcb90958523a1f6f2c4a6af8bdc"
Nov 01 09:17:02.387117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:02.387085    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:03.996146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:03.996108    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"]
Nov 01 09:17:03.996691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:03.996674    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7a7c4d40-b881-45d2-84c0-e16becfca818" containerName="revoke-keys"
Nov 01 09:17:03.996745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:03.996692    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7c4d40-b881-45d2-84c0-e16becfca818" containerName="revoke-keys"
Nov 01 09:17:03.996797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:03.996788    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7c4d40-b881-45d2-84c0-e16becfca818" containerName="revoke-keys"
Nov 01 09:17:04.000478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.000462    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.095627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.095595    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:17:04.095827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.095699    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:17:04.095827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.095700    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:17:04.095827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.095759    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:17:04.099309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.099290    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"]
Nov 01 09:17:04.107893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.107871    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbv4\" (UniqueName: \"kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.107958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.107903    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.108111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.108092    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.208723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.208693    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.208859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.208738    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbv4\" (UniqueName: \"kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.208859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.208770    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.209335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.209313    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.211166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.211142    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.266150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.266074    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbv4\" (UniqueName: \"kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4\") pod \"maas-api-revoke-sub-60ff1694-7hxwn\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.321476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.321449    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:04.610157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.610053    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:04.694695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.694669    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"]
Nov 01 09:17:04.695446 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:04.695394    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df58d2c_fd43_499f_ab6c_e6eb4fb21967.slice/crio-735a94f2c60409d67701d34e6bf1aee14dcf50d744058bf3050d710c576a83ae WatchSource:0}: Error finding container 735a94f2c60409d67701d34e6bf1aee14dcf50d744058bf3050d710c576a83ae: Status 404 returned error can't find the container with id 735a94f2c60409d67701d34e6bf1aee14dcf50d744058bf3050d710c576a83ae
Nov 01 09:17:04.754192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:04.754165    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:05.503281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:05.503245    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn" event={"ID":"2df58d2c-fd43-499f-ab6c-e6eb4fb21967","Type":"ContainerStarted","Data":"36a1a4ab8ab50558f8be1e54c393ba38bd5b07a0f500927001566ee030170e98"}
Nov 01 09:17:05.503820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:05.503296    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn" event={"ID":"2df58d2c-fd43-499f-ab6c-e6eb4fb21967","Type":"ContainerStarted","Data":"735a94f2c60409d67701d34e6bf1aee14dcf50d744058bf3050d710c576a83ae"}
Nov 01 09:17:05.614991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:05.614941    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn" podStartSLOduration=2.6149274350000002 podStartE2EDuration="2.614927435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:05.610752806 +0000 UTC m=+2698.805147365" watchObservedRunningTime="2026-11-01 09:17:05.614927435 +0000 UTC m=+2698.809321991"
Nov 01 09:17:06.205960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:06.205922    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:06.510974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:06.510937    4314 generic.go:356] "Generic (PLEG): container finished" podID="2df58d2c-fd43-499f-ab6c-e6eb4fb21967" containerID="36a1a4ab8ab50558f8be1e54c393ba38bd5b07a0f500927001566ee030170e98" exitCode=0
Nov 01 09:17:06.511372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:06.511017    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn" event={"ID":"2df58d2c-fd43-499f-ab6c-e6eb4fb21967","Type":"ContainerDied","Data":"36a1a4ab8ab50558f8be1e54c393ba38bd5b07a0f500927001566ee030170e98"}
Nov 01 09:17:06.711609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:06.711576    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:07.182098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.182062    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"]
Nov 01 09:17:07.186011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.185996    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.245928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.245894    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfx8w\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.245938    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.245971    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.246116    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.246171    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.246244    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.246303    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.246383    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.246462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.246449    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.286044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.286007    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"]
Nov 01 09:17:07.347205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347170    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347259    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347336    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347367    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347430    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347461    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347577    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347609    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfx8w\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347645    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347909    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.347997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.347959    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.348183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.348160    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.348243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.348187    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.348459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.348382    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.349882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.349858    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.350352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.350328    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.405098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.405061    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.405280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.405178    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfx8w\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.434223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.434141    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:07.500459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.500425    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:07.694330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.694270    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:07.752236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.752204    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca\") pod \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") "
Nov 01 09:17:07.752400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.752295    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4\" (UniqueName: \"kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4\") pod \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") "
Nov 01 09:17:07.752445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.752428    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca\") pod \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\" (UID: \"2df58d2c-fd43-499f-ab6c-e6eb4fb21967\") "
Nov 01 09:17:07.752655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.752630    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca" pod "2df58d2c-fd43-499f-ab6c-e6eb4fb21967" (UID: "2df58d2c-fd43-499f-ab6c-e6eb4fb21967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:07.752850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.752828    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:07.754395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.754376    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca" pod "2df58d2c-fd43-499f-ab6c-e6eb4fb21967" (UID: "2df58d2c-fd43-499f-ab6c-e6eb4fb21967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:17:07.754466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.754446    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4" pod "2df58d2c-fd43-499f-ab6c-e6eb4fb21967" (UID: "2df58d2c-fd43-499f-ab6c-e6eb4fb21967"). InnerVolumeSpecName "kube-api-access-hwbv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:07.854457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.854427    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbv4\" (UniqueName: \"kubernetes.io/projected/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-kube-api-access-hwbv4\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:07.854457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.854452    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df58d2c-fd43-499f-ab6c-e6eb4fb21967-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:07.960842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.960811    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"]
Nov 01 09:17:07.961564 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:07.961520    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb42d12b_b56d_448b_8087_94a155a7b73d.slice/crio-db1934b3391ea580f78077596a6f841c3d9f357de657b7c31ec2cfd4567ea0c7 WatchSource:0}: Error finding container db1934b3391ea580f78077596a6f841c3d9f357de657b7c31ec2cfd4567ea0c7: Status 404 returned error can't find the container with id db1934b3391ea580f78077596a6f841c3d9f357de657b7c31ec2cfd4567ea0c7
Nov 01 09:17:07.964053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.964017    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:07.964135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.964086    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:07.964135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:07.964122    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:08.526324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:08.526279    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" event={"ID":"eb42d12b-b56d-448b-8087-94a155a7b73d","Type":"ContainerStarted","Data":"bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d"}
Nov 01 09:17:08.526513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:08.526350    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" event={"ID":"eb42d12b-b56d-448b-8087-94a155a7b73d","Type":"ContainerStarted","Data":"db1934b3391ea580f78077596a6f841c3d9f357de657b7c31ec2cfd4567ea0c7"}
Nov 01 09:17:08.527669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:08.527646    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"
Nov 01 09:17:08.527669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:08.527655    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn" event={"ID":"2df58d2c-fd43-499f-ab6c-e6eb4fb21967","Type":"ContainerDied","Data":"735a94f2c60409d67701d34e6bf1aee14dcf50d744058bf3050d710c576a83ae"}
Nov 01 09:17:08.527844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:08.527696    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735a94f2c60409d67701d34e6bf1aee14dcf50d744058bf3050d710c576a83ae"
Nov 01 09:17:08.651568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:08.651451    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podStartSLOduration=1.651435896 podStartE2EDuration="1.651435896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:08.647382388 +0000 UTC m=+2701.841776934" watchObservedRunningTime="2026-11-01 09:17:08.651435896 +0000 UTC m=+2701.845830450"
Nov 01 09:17:09.500837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:09.500801    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:09.502589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:09.502560    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:17:09.502743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:09.502616    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:17:10.500950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:10.500904    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:17:10.501430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:10.500970    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:17:11.323320 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:17:11.323286    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:17:11.501822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:11.501787    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:17:11.502222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:11.501840    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:17:12.225486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:12.225457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:12.501160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:12.501123    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:17:12.501349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:12.501189    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:17:12.515124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:12.515093    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:13.501725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:13.501687    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:17:13.501915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:13.501746    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:17:14.501059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:14.501017    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:17:14.501638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:14.501103    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:17:15.333665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:15.333627    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:15.950990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:15.950953    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:15.951639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:15.951601    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:15.952290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:15.952272    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:17:17.314579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:17.314522    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:18.485573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:18.485521    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:19.389864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:19.389830    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"]
Nov 01 09:17:19.496881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:19.496849    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"]
Nov 01 09:17:19.744318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:19.744281    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:19.896558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:19.893552    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:22.290053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.290015    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"]
Nov 01 09:17:22.291403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.291374    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2df58d2c-fd43-499f-ab6c-e6eb4fb21967" containerName="revoke-keys"
Nov 01 09:17:22.291566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.291550    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df58d2c-fd43-499f-ab6c-e6eb4fb21967" containerName="revoke-keys"
Nov 01 09:17:22.291838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.291821    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df58d2c-fd43-499f-ab6c-e6eb4fb21967" containerName="revoke-keys"
Nov 01 09:17:22.296989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.296969    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.335113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.335091    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"]
Nov 01 09:17:22.339102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.339086    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.390413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.390368    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"]
Nov 01 09:17:22.439637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.439604    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"]
Nov 01 09:17:22.467439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467399    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.467656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467448    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.467656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467586    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.467656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467646    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.467856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467675    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.467856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467713    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.467856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467743    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.468026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467888    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.468026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467934    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.468026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.467982    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4v5\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.468140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468035    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.468175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468129    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnlp\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.468271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468251    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.468337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468324    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.468400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468373    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.468453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468407    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.468500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468454    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.468500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.468486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.505298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.505263    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:22.569851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.569756    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4v5\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.569851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.569811    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.569860    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnlp\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.569949    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.569987    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570021    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570052    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570281    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570342    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570352    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570389    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570428    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570481    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570483    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570565    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570593    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570631    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570649    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570665    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.570865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570711    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.570865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.570752    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.571371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.571344    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.571577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.571368    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.571923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.571686    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.571923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.571558    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.571923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.571660    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.572452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.572415    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.572630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.572617    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.572811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.572784    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.573289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.573268    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.573623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.573605    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.573874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.573856    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.627318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.627280    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.627487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.627296    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.627487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.627472    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnlp\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.627619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.627569    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4v5\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:22.652733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.652710    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:22.910406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:22.910322    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:23.072153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.072125    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"]
Nov 01 09:17:23.072514 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:23.072488    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6f95e2_4dca_4643_a8b2_832293ad7d86.slice/crio-5d52a05b52127cf01f2d98960a9c9a4ffa493449d4fd6d90496c7656084b2e85 WatchSource:0}: Error finding container 5d52a05b52127cf01f2d98960a9c9a4ffa493449d4fd6d90496c7656084b2e85: Status 404 returned error can't find the container with id 5d52a05b52127cf01f2d98960a9c9a4ffa493449d4fd6d90496c7656084b2e85
Nov 01 09:17:23.075024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.074992    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:23.075103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.075057    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:23.075103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.075089    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:23.109026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.108998    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"]
Nov 01 09:17:23.114306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.114276    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.165923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.165830    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:17:23.166137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.165960    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:17:23.166137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.166015    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:17:23.166137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.166072    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:17:23.167982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.167956    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"]
Nov 01 09:17:23.281180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.281145    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7j9\" (UniqueName: \"kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.281356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.281200    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.281356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.281341    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.289370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.289337    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"]
Nov 01 09:17:23.289819 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:23.289794    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dae45d_623b_4f22_8039_36d33f892acf.slice/crio-3ddd976b87cd3249e6ea7b47c5cc57a10e9e4ebe2e1c3c6f149263da19c9282d WatchSource:0}: Error finding container 3ddd976b87cd3249e6ea7b47c5cc57a10e9e4ebe2e1c3c6f149263da19c9282d: Status 404 returned error can't find the container with id 3ddd976b87cd3249e6ea7b47c5cc57a10e9e4ebe2e1c3c6f149263da19c9282d
Nov 01 09:17:23.292355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.292326    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:23.292666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.292383    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:23.292666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.292411    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:23.382817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.382785    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7j9\" (UniqueName: \"kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.382942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.382861    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.383023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.383005    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.383488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.383464    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.385343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.385323    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.440321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.440010    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7j9\" (UniqueName: \"kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9\") pod \"maas-api-revoke-sub-4aaec97f-bpj8x\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.672486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.672447    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" event={"ID":"bb6f95e2-4dca-4643-a8b2-832293ad7d86","Type":"ContainerStarted","Data":"2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215"}
Nov 01 09:17:23.672729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.672507    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" event={"ID":"bb6f95e2-4dca-4643-a8b2-832293ad7d86","Type":"ContainerStarted","Data":"5d52a05b52127cf01f2d98960a9c9a4ffa493449d4fd6d90496c7656084b2e85"}
Nov 01 09:17:23.674793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.674761    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" event={"ID":"16dae45d-623b-4f22-8039-36d33f892acf","Type":"ContainerStarted","Data":"71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686"}
Nov 01 09:17:23.674923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.674815    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" event={"ID":"16dae45d-623b-4f22-8039-36d33f892acf","Type":"ContainerStarted","Data":"3ddd976b87cd3249e6ea7b47c5cc57a10e9e4ebe2e1c3c6f149263da19c9282d"}
Nov 01 09:17:23.731939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.731902    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:23.789743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.789687    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podStartSLOduration=1.789668862 podStartE2EDuration="1.789668862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:23.784737823 +0000 UTC m=+2716.979132407" watchObservedRunningTime="2026-11-01 09:17:23.789668862 +0000 UTC m=+2716.984063419"
Nov 01 09:17:23.902218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.902159    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podStartSLOduration=1.902145466 podStartE2EDuration="1.902145466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:23.897621352 +0000 UTC m=+2717.092015920" watchObservedRunningTime="2026-11-01 09:17:23.902145466 +0000 UTC m=+2717.096540021"
Nov 01 09:17:23.911331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:23.911302    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:24.110725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.110697    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"]
Nov 01 09:17:24.111591 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:24.111561    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcf6b0d_0549_4dd3_a057_5d9411b63883.slice/crio-034ee35dc47228c809aec3ea52a7b7afa36ff101337ba8fb1dabc2df704d6795 WatchSource:0}: Error finding container 034ee35dc47228c809aec3ea52a7b7afa36ff101337ba8fb1dabc2df704d6795: Status 404 returned error can't find the container with id 034ee35dc47228c809aec3ea52a7b7afa36ff101337ba8fb1dabc2df704d6795
Nov 01 09:17:24.393723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.393610    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69" podUID="04e1e614-7c20-4e13-a58c-330f768c6867" containerName="payload-processing" containerID="cri-o://238cf4745bb10475527ed047bb5191ad276c323db299779a3c55357f2ec6226b" gracePeriod=25
Nov 01 09:17:24.417740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.417712    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:24.508817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.508685    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4" podUID="73d957b9-3510-48b0-b31f-372e9f55f508" containerName="payload-pre-processing" containerID="cri-o://0a9030f7131219a927cafa3f4c4bc8143a3bf230e9fd3151b1c85857e737165a" gracePeriod=25
Nov 01 09:17:24.653905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.653827    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:24.655560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.655226    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 01 09:17:24.655560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.655279    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 01 09:17:24.728191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.728143    4314 generic.go:356] "Generic (PLEG): container finished" podID="73d957b9-3510-48b0-b31f-372e9f55f508" containerID="0a9030f7131219a927cafa3f4c4bc8143a3bf230e9fd3151b1c85857e737165a" exitCode=0
Nov 01 09:17:24.728410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.728314    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4" event={"ID":"73d957b9-3510-48b0-b31f-372e9f55f508","Type":"ContainerDied","Data":"0a9030f7131219a927cafa3f4c4bc8143a3bf230e9fd3151b1c85857e737165a"}
Nov 01 09:17:24.730154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.730126    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x" event={"ID":"bdcf6b0d-0549-4dd3-a057-5d9411b63883","Type":"ContainerStarted","Data":"b5e29ae580e28808bf477fff03ac2a472f7d7a5b1286fa0a9edf421375638f13"}
Nov 01 09:17:24.730276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.730172    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x" event={"ID":"bdcf6b0d-0549-4dd3-a057-5d9411b63883","Type":"ContainerStarted","Data":"034ee35dc47228c809aec3ea52a7b7afa36ff101337ba8fb1dabc2df704d6795"}
Nov 01 09:17:24.734768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.733416    4314 generic.go:356] "Generic (PLEG): container finished" podID="04e1e614-7c20-4e13-a58c-330f768c6867" containerID="238cf4745bb10475527ed047bb5191ad276c323db299779a3c55357f2ec6226b" exitCode=0
Nov 01 09:17:24.734768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.734242    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69" event={"ID":"04e1e614-7c20-4e13-a58c-330f768c6867","Type":"ContainerDied","Data":"238cf4745bb10475527ed047bb5191ad276c323db299779a3c55357f2ec6226b"}
Nov 01 09:17:24.850750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.850689    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x" podStartSLOduration=1.850669227 podStartE2EDuration="1.850669227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:24.844969805 +0000 UTC m=+2718.039364360" watchObservedRunningTime="2026-11-01 09:17:24.850669227 +0000 UTC m=+2718.045063782"
Nov 01 09:17:24.852784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.852758    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:17:24.916958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.914348    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:24.916958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:24.914400    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:25.021657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.021603    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn\" (UniqueName: \"kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn\") pod \"04e1e614-7c20-4e13-a58c-330f768c6867\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") "
Nov 01 09:17:25.021998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.021971    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume\") pod \"04e1e614-7c20-4e13-a58c-330f768c6867\" (UID: \"04e1e614-7c20-4e13-a58c-330f768c6867\") "
Nov 01 09:17:25.023069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.023045    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume" pod "04e1e614-7c20-4e13-a58c-330f768c6867" (UID: "04e1e614-7c20-4e13-a58c-330f768c6867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:25.026606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.026584    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn" pod "04e1e614-7c20-4e13-a58c-330f768c6867" (UID: "04e1e614-7c20-4e13-a58c-330f768c6867"). InnerVolumeSpecName "kube-api-access-vc7sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:25.031048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.031013    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:17:25.124042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.124007    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e1e614-7c20-4e13-a58c-330f768c6867-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:25.124042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.124038    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7sn\" (UniqueName: \"kubernetes.io/projected/04e1e614-7c20-4e13-a58c-330f768c6867-kube-api-access-vc7sn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:25.225835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.225762    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume\") pod \"73d957b9-3510-48b0-b31f-372e9f55f508\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") "
Nov 01 09:17:25.226048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.226034    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6\" (UniqueName: \"kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6\") pod \"73d957b9-3510-48b0-b31f-372e9f55f508\" (UID: \"73d957b9-3510-48b0-b31f-372e9f55f508\") "
Nov 01 09:17:25.226674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.226643    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume" pod "73d957b9-3510-48b0-b31f-372e9f55f508" (UID: "73d957b9-3510-48b0-b31f-372e9f55f508"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:25.228768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.228748    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6" pod "73d957b9-3510-48b0-b31f-372e9f55f508" (UID: "73d957b9-3510-48b0-b31f-372e9f55f508"). InnerVolumeSpecName "kube-api-access-d8fm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:25.315915 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:17:25.315874    4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcf6b0d_0549_4dd3_a057_5d9411b63883.slice/crio-conmon-b5e29ae580e28808bf477fff03ac2a472f7d7a5b1286fa0a9edf421375638f13.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 09:17:25.327261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.327227    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d957b9-3510-48b0-b31f-372e9f55f508-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:25.327261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.327259    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8fm6\" (UniqueName: \"kubernetes.io/projected/73d957b9-3510-48b0-b31f-372e9f55f508-kube-api-access-d8fm6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:25.653579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.653523    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 01 09:17:25.654006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.653595    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 01 09:17:25.741581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.741454    4314 generic.go:356] "Generic (PLEG): container finished" podID="bdcf6b0d-0549-4dd3-a057-5d9411b63883" containerID="b5e29ae580e28808bf477fff03ac2a472f7d7a5b1286fa0a9edf421375638f13" exitCode=0
Nov 01 09:17:25.741581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.741550    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x" event={"ID":"bdcf6b0d-0549-4dd3-a057-5d9411b63883","Type":"ContainerDied","Data":"b5e29ae580e28808bf477fff03ac2a472f7d7a5b1286fa0a9edf421375638f13"}
Nov 01 09:17:25.742958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.742938    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"
Nov 01 09:17:25.743105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.742971    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69" event={"ID":"04e1e614-7c20-4e13-a58c-330f768c6867","Type":"ContainerDied","Data":"773d06e3a17c7a6702f5008630a140af1872c6588fe4724d7c1b373f62668e17"}
Nov 01 09:17:25.743105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.743007    4314 scope.go:122] "RemoveContainer" containerID="238cf4745bb10475527ed047bb5191ad276c323db299779a3c55357f2ec6226b"
Nov 01 09:17:25.744493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.744466    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4" event={"ID":"73d957b9-3510-48b0-b31f-372e9f55f508","Type":"ContainerDied","Data":"282e39eb2ef6a3c008e3b217dc8fa72933fea02e9aa2340701ee379988fc6080"}
Nov 01 09:17:25.744493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.744487    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"
Nov 01 09:17:25.753952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.753934    4314 scope.go:122] "RemoveContainer" containerID="0a9030f7131219a927cafa3f4c4bc8143a3bf230e9fd3151b1c85857e737165a"
Nov 01 09:17:25.912614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.911733    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:25.912614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.911798    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:26.010728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:25.983594    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"]
Nov 01 09:17:26.010728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.010634    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1c173-79bf9c64fc-8pl69"]
Nov 01 09:17:26.109589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.109462    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"]
Nov 01 09:17:26.114163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.114137    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1c173-69f548f687-kzfz4"]
Nov 01 09:17:26.653470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.653433    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 01 09:17:26.653650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.653486    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 01 09:17:26.914201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.913643    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:26.914201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.913713    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:26.934267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.934224    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e1e614-7c20-4e13-a58c-330f768c6867" path="/var/lib/kubelet/pods/04e1e614-7c20-4e13-a58c-330f768c6867/volumes"
Nov 01 09:17:26.934866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:26.934843    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d957b9-3510-48b0-b31f-372e9f55f508" path="/var/lib/kubelet/pods/73d957b9-3510-48b0-b31f-372e9f55f508/volumes"
Nov 01 09:17:27.045804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.045779    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:27.144849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.144815    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9\" (UniqueName: \"kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9\") pod \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") "
Nov 01 09:17:27.145137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.145120    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca\") pod \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") "
Nov 01 09:17:27.145269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.145256    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca\") pod \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\" (UID: \"bdcf6b0d-0549-4dd3-a057-5d9411b63883\") "
Nov 01 09:17:27.145590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.145566    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca" pod "bdcf6b0d-0549-4dd3-a057-5d9411b63883" (UID: "bdcf6b0d-0549-4dd3-a057-5d9411b63883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:27.145783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.145767    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:27.147020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.146993    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca" pod "bdcf6b0d-0549-4dd3-a057-5d9411b63883" (UID: "bdcf6b0d-0549-4dd3-a057-5d9411b63883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:17:27.147118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.147076    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9" pod "bdcf6b0d-0549-4dd3-a057-5d9411b63883" (UID: "bdcf6b0d-0549-4dd3-a057-5d9411b63883"). InnerVolumeSpecName "kube-api-access-qq7j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:27.246992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.246960    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7j9\" (UniqueName: \"kubernetes.io/projected/bdcf6b0d-0549-4dd3-a057-5d9411b63883-kube-api-access-qq7j9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:27.246992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.246997    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf6b0d-0549-4dd3-a057-5d9411b63883-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:27.422974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.422911    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:27.653276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.653189    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 01 09:17:27.653276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.653242    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 01 09:17:27.812721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.812683    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x" event={"ID":"bdcf6b0d-0549-4dd3-a057-5d9411b63883","Type":"ContainerDied","Data":"034ee35dc47228c809aec3ea52a7b7afa36ff101337ba8fb1dabc2df704d6795"}
Nov 01 09:17:27.813234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.812734    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="034ee35dc47228c809aec3ea52a7b7afa36ff101337ba8fb1dabc2df704d6795"
Nov 01 09:17:27.813234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.812789    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"
Nov 01 09:17:27.911084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.910981    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:27.911084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:27.911040    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:28.653159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:28.653120    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 01 09:17:28.653345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:28.653172    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 01 09:17:28.911367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:28.911273    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:28.912013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:28.911970    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:29.653849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:29.653808    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 01 09:17:29.654015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:29.653877    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 01 09:17:29.911609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:29.911492    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:29.911609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:29.911566    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:30.911243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:30.911200    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 01 09:17:30.911460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:30.911263    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 01 09:17:31.221213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:31.221132    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:31.222468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:31.222442    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:31.223438 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:31.223414    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:17:31.915598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:31.915565    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:31.915866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:31.915844    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:31.916473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:31.916456    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:17:32.344801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:32.344742    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:34.175177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:34.175147    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"]
Nov 01 09:17:34.175667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:34.175372    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy" containerID="cri-o://cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e" gracePeriod=30
Nov 01 09:17:37.084265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:37.084228    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:38.046896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.046854    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"]
Nov 01 09:17:38.092491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.092452    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64426461-cq5gm"]
Nov 01 09:17:38.302748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.302665    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"]
Nov 01 09:17:38.303183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303171    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bdcf6b0d-0549-4dd3-a057-5d9411b63883" containerName="revoke-keys"
Nov 01 09:17:38.303228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303185    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcf6b0d-0549-4dd3-a057-5d9411b63883" containerName="revoke-keys"
Nov 01 09:17:38.303228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303199    4314 cpu_manager.go:414] "containerMap: removing container" podUID="73d957b9-3510-48b0-b31f-372e9f55f508" containerName="payload-pre-processing"
Nov 01 09:17:38.303228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303205    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d957b9-3510-48b0-b31f-372e9f55f508" containerName="payload-pre-processing"
Nov 01 09:17:38.303228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303213    4314 cpu_manager.go:414] "containerMap: removing container" podUID="04e1e614-7c20-4e13-a58c-330f768c6867" containerName="payload-processing"
Nov 01 09:17:38.303228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303218    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e1e614-7c20-4e13-a58c-330f768c6867" containerName="payload-processing"
Nov 01 09:17:38.303392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303339    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcf6b0d-0549-4dd3-a057-5d9411b63883" containerName="revoke-keys"
Nov 01 09:17:38.303392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303349    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d957b9-3510-48b0-b31f-372e9f55f508" containerName="payload-pre-processing"
Nov 01 09:17:38.303392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.303355    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e1e614-7c20-4e13-a58c-330f768c6867" containerName="payload-processing"
Nov 01 09:17:38.306777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.306755    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.358717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.358674    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:17:38.358933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.358753    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:17:38.358933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.358842    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:17:38.359041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.358938    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:17:38.359807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.359786    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"]
Nov 01 09:17:38.430100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.430071    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.430262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.430114    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.430310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.430266    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rht6k\" (UniqueName: \"kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.531006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.530972    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.531198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.531046    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.531284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.531200    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rht6k\" (UniqueName: \"kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.531826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.531799    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.533704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.533681    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.588216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.588146    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rht6k\" (UniqueName: \"kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k\") pod \"maas-api-revoke-sub-0fe12058-s8c49\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.620034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.620003    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:38.928445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.928353    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c518986-7101-443a-9334-c7a10b36ffb3" path="/var/lib/kubelet/pods/1c518986-7101-443a-9334-c7a10b36ffb3/volumes"
Nov 01 09:17:38.989334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:38.989309    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"]
Nov 01 09:17:38.990796 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:38.990767    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176114b6_8039_496b_9435_6b83d642bd64.slice/crio-baadad55afabc837d8a3c412ac9da9d446f817bbd6137135f79028c8580b5333 WatchSource:0}: Error finding container baadad55afabc837d8a3c412ac9da9d446f817bbd6137135f79028c8580b5333: Status 404 returned error can't find the container with id baadad55afabc837d8a3c412ac9da9d446f817bbd6137135f79028c8580b5333
Nov 01 09:17:39.408948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.408923    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 09:17:39.542677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.542642    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.542863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.542768    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.542953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.542860    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.542962    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.542965    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:17:39.543018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543006    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543076    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543104    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543143    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543173    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs\") pod \"8fac6601-db7c-4e32-84fc-5920fa3979b8\" (UID: \"8fac6601-db7c-4e32-84fc-5920fa3979b8\") "
Nov 01 09:17:39.543419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543359    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:17:39.543580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543558    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:17:39.543818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543723    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.543818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543777    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.543818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543792    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.543818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.543791    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:17:39.544600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.544303    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:39.546288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.546045    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "kube-api-access-m4b2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:39.546392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.546307    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:17:39.546392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.546314    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:39.546518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.546442    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo" pod "8fac6601-db7c-4e32-84fc-5920fa3979b8" (UID: "8fac6601-db7c-4e32-84fc-5920fa3979b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:17:39.644504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.644475    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.644504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.644502    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.644504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.644511    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.644754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.644520    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac6601-db7c-4e32-84fc-5920fa3979b8-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.644754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.644528    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.644754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.644548    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4b2x\" (UniqueName: \"kubernetes.io/projected/8fac6601-db7c-4e32-84fc-5920fa3979b8-kube-api-access-m4b2x\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:39.899944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.899859    4314 generic.go:356] "Generic (PLEG): container finished" podID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerID="cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e" exitCode=0
Nov 01 09:17:39.899944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.899926    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"
Nov 01 09:17:39.900201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.899948    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" event={"ID":"8fac6601-db7c-4e32-84fc-5920fa3979b8","Type":"ContainerDied","Data":"cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e"}
Nov 01 09:17:39.900201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.900041    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9" event={"ID":"8fac6601-db7c-4e32-84fc-5920fa3979b8","Type":"ContainerDied","Data":"a12f77995e2973c0e1a0a6df3be4ac55e1b519f417b7525d1fb08d0a04a9ba13"}
Nov 01 09:17:39.900201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.900077    4314 scope.go:122] "RemoveContainer" containerID="cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e"
Nov 01 09:17:39.901867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.901843    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49" event={"ID":"176114b6-8039-496b-9435-6b83d642bd64","Type":"ContainerStarted","Data":"72975d332216b4c65df802c1b8105fae4d125dae401be556678dca241aedbff6"}
Nov 01 09:17:39.901998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.901882    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49" event={"ID":"176114b6-8039-496b-9435-6b83d642bd64","Type":"ContainerStarted","Data":"baadad55afabc837d8a3c412ac9da9d446f817bbd6137135f79028c8580b5333"}
Nov 01 09:17:39.910678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.910662    4314 scope.go:122] "RemoveContainer" containerID="41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"
Nov 01 09:17:39.919965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.919948    4314 scope.go:122] "RemoveContainer" containerID="cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e"
Nov 01 09:17:39.920224 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:17:39.920205    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e\": container with ID starting with cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e not found: ID does not exist" containerID="cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e"
Nov 01 09:17:39.920270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.920232    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e"} err="failed to get container status \"cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e\": rpc error: code = NotFound desc = could not find container \"cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e\": container with ID starting with cd0846ec96ad8fed04079c00bffc686b37a9e9e2bd7c46dcf633ad8e1d26448e not found: ID does not exist"
Nov 01 09:17:39.920270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.920249    4314 scope.go:122] "RemoveContainer" containerID="41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"
Nov 01 09:17:39.920518 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:17:39.920498    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945\": container with ID starting with 41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945 not found: ID does not exist" containerID="41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"
Nov 01 09:17:39.920576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:39.920525    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945"} err="failed to get container status \"41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945\": rpc error: code = NotFound desc = could not find container \"41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945\": container with ID starting with 41f695e581f587cb7d01d1650ade8a5a42cf6b91b2b5cb2b5c0800614a6b6945 not found: ID does not exist"
Nov 01 09:17:40.054993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:40.054961    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"]
Nov 01 09:17:40.061669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:40.061642    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-sqkz9"]
Nov 01 09:17:40.167173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:40.167075    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49" podStartSLOduration=2.167060683 podStartE2EDuration="2.167060683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:40.16465367 +0000 UTC m=+2733.359048237" watchObservedRunningTime="2026-11-01 09:17:40.167060683 +0000 UTC m=+2733.361455238"
Nov 01 09:17:40.910252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:40.910212    4314 generic.go:356] "Generic (PLEG): container finished" podID="176114b6-8039-496b-9435-6b83d642bd64" containerID="72975d332216b4c65df802c1b8105fae4d125dae401be556678dca241aedbff6" exitCode=0
Nov 01 09:17:40.910702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:40.910291    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49" event={"ID":"176114b6-8039-496b-9435-6b83d642bd64","Type":"ContainerDied","Data":"72975d332216b4c65df802c1b8105fae4d125dae401be556678dca241aedbff6"}
Nov 01 09:17:40.932188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:40.932150    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" path="/var/lib/kubelet/pods/8fac6601-db7c-4e32-84fc-5920fa3979b8/volumes"
Nov 01 09:17:41.323115 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:17:41.323079    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:17:41.970678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:41.970645    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:42.072473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.072444    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:42.175972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.175887    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca\") pod \"176114b6-8039-496b-9435-6b83d642bd64\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") "
Nov 01 09:17:42.176143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.176015    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca\") pod \"176114b6-8039-496b-9435-6b83d642bd64\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") "
Nov 01 09:17:42.176143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.176079    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k\" (UniqueName: \"kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k\") pod \"176114b6-8039-496b-9435-6b83d642bd64\" (UID: \"176114b6-8039-496b-9435-6b83d642bd64\") "
Nov 01 09:17:42.176429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.176405    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca" pod "176114b6-8039-496b-9435-6b83d642bd64" (UID: "176114b6-8039-496b-9435-6b83d642bd64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:42.177983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.177958    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca" pod "176114b6-8039-496b-9435-6b83d642bd64" (UID: "176114b6-8039-496b-9435-6b83d642bd64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:17:42.178060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.178008    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k" pod "176114b6-8039-496b-9435-6b83d642bd64" (UID: "176114b6-8039-496b-9435-6b83d642bd64"). InnerVolumeSpecName "kube-api-access-rht6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:42.278015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.277980    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176114b6-8039-496b-9435-6b83d642bd64-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:42.278015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.278011    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176114b6-8039-496b-9435-6b83d642bd64-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:42.278015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.278023    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rht6k\" (UniqueName: \"kubernetes.io/projected/176114b6-8039-496b-9435-6b83d642bd64-kube-api-access-rht6k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:42.925303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.925276    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"
Nov 01 09:17:42.932166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.932137    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49" event={"ID":"176114b6-8039-496b-9435-6b83d642bd64","Type":"ContainerDied","Data":"baadad55afabc837d8a3c412ac9da9d446f817bbd6137135f79028c8580b5333"}
Nov 01 09:17:42.932357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:42.932184    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baadad55afabc837d8a3c412ac9da9d446f817bbd6137135f79028c8580b5333"
Nov 01 09:17:45.204543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:45.204507    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:46.277560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.274102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:46.425559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.425058    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:46.826723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.826684    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"]
Nov 01 09:17:46.831918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.831878    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy"
Nov 01 09:17:46.832168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.832150    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy"
Nov 01 09:17:46.832355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.832340    4314 cpu_manager.go:414] "containerMap: removing container" podUID="176114b6-8039-496b-9435-6b83d642bd64" containerName="revoke-keys"
Nov 01 09:17:46.832472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.832459    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176114b6-8039-496b-9435-6b83d642bd64" containerName="revoke-keys"
Nov 01 09:17:46.832774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.832756    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="176114b6-8039-496b-9435-6b83d642bd64" containerName="revoke-keys"
Nov 01 09:17:46.832913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.832900    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy"
Nov 01 09:17:46.833014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.833000    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy"
Nov 01 09:17:46.852072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.852040    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:46.881792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.878065    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:47.006563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:46.941632    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3b7524-openshift-default-dockercfg-vmprg\""
Nov 01 09:17:47.006563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006427    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006517    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006569    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006629    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006713    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006766    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006836    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006885    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwkg\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.006935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.006913    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.109153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.109062    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.109876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.109857    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.110832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.110810    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.111282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111266    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwkg\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.111414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111401    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.111519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111507    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.111701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111686    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.111809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111798    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.111952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111937    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.112852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.112820    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.113001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.109743    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.113316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.111203    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.113833 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.113812    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.128549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.114974    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.131369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.131337    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.158050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.158018    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"]
Nov 01 09:17:47.180587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.173183    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.180587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.173317    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:47.192890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.192814    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwkg\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.233564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.211090    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token\") pod \"e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") " pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.430208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.429487    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:47.501882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.501848    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:47.583563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:47.583434    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:48.060959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.060862    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"]
Nov 01 09:17:48.061424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.061406    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy"
Nov 01 09:17:48.061424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.061424    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fac6601-db7c-4e32-84fc-5920fa3979b8" containerName="istio-proxy"
Nov 01 09:17:48.066183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.066158    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.114886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.114850    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:17:48.115146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.114943    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:17:48.115873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.115662    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:17:48.115873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.115794    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:17:48.121278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.121253    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"]
Nov 01 09:17:48.153509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.153477    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvcf\" (UniqueName: \"kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.153700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.153661    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.153756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.153740    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.215216 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:48.215187    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bc7c4b_b4b6_43ba_a76e_74ecfb931909.slice/crio-1b0ad9a0bab3f7f404bcba3f5ac72d4538f5844644d670e8a2c3c347bb9d0164 WatchSource:0}: Error finding container 1b0ad9a0bab3f7f404bcba3f5ac72d4538f5844644d670e8a2c3c347bb9d0164: Status 404 returned error can't find the container with id 1b0ad9a0bab3f7f404bcba3f5ac72d4538f5844644d670e8a2c3c347bb9d0164
Nov 01 09:17:48.215457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.215431    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"]
Nov 01 09:17:48.218054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.218013    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:48.218169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.218096    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:48.218169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.218137    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:17:48.255615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.255589    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.255755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.255718    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvcf\" (UniqueName: \"kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.255799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.255787    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.256388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.256327    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.258163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.258143    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.312823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.312758    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvcf\" (UniqueName: \"kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf\") pod \"maas-api-revoke-sub-c843d5ae-xpqt8\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.382152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.382113    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:48.752784 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:17:48.752757    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575decce_06ba_40bf_9e56_db6982769f37.slice/crio-4e0738fea2bcd14c119589723747849e7db716e99511ebc4bc1e57e743758f94 WatchSource:0}: Error finding container 4e0738fea2bcd14c119589723747849e7db716e99511ebc4bc1e57e743758f94: Status 404 returned error can't find the container with id 4e0738fea2bcd14c119589723747849e7db716e99511ebc4bc1e57e743758f94
Nov 01 09:17:48.753716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:48.753643    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"]
Nov 01 09:17:49.035364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:49.035279    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" event={"ID":"85bc7c4b-b4b6-43ba-a76e-74ecfb931909","Type":"ContainerStarted","Data":"97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"}
Nov 01 09:17:49.035364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:49.035342    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" event={"ID":"85bc7c4b-b4b6-43ba-a76e-74ecfb931909","Type":"ContainerStarted","Data":"1b0ad9a0bab3f7f404bcba3f5ac72d4538f5844644d670e8a2c3c347bb9d0164"}
Nov 01 09:17:49.036704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:49.036674    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8" event={"ID":"575decce-06ba-40bf-9e56-db6982769f37","Type":"ContainerStarted","Data":"4e0738fea2bcd14c119589723747849e7db716e99511ebc4bc1e57e743758f94"}
Nov 01 09:17:49.178334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:49.178266    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podStartSLOduration=3.178253377 podStartE2EDuration="3.178253377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:49.176240034 +0000 UTC m=+2742.370634590" watchObservedRunningTime="2026-11-01 09:17:49.178253377 +0000 UTC m=+2742.372647932"
Nov 01 09:17:49.503076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:49.503043    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:49.508558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:49.508506    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:50.045948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:50.045909    4314 generic.go:356] "Generic (PLEG): container finished" podID="575decce-06ba-40bf-9e56-db6982769f37" containerID="94bd5557b922c70ddc876cb6d0a8bd949bb5b1a81dc681e1d05aaee590b4afae" exitCode=0
Nov 01 09:17:50.046432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:50.045992    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8" event={"ID":"575decce-06ba-40bf-9e56-db6982769f37","Type":"ContainerDied","Data":"94bd5557b922c70ddc876cb6d0a8bd949bb5b1a81dc681e1d05aaee590b4afae"}
Nov 01 09:17:50.046432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:50.046365    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:50.047192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:50.047173    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:17:51.186373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.186344    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:51.293403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.293369    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca\") pod \"575decce-06ba-40bf-9e56-db6982769f37\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") "
Nov 01 09:17:51.293612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.293459    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf\" (UniqueName: \"kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf\") pod \"575decce-06ba-40bf-9e56-db6982769f37\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") "
Nov 01 09:17:51.293612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.293517    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca\") pod \"575decce-06ba-40bf-9e56-db6982769f37\" (UID: \"575decce-06ba-40bf-9e56-db6982769f37\") "
Nov 01 09:17:51.294349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.293953    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca" pod "575decce-06ba-40bf-9e56-db6982769f37" (UID: "575decce-06ba-40bf-9e56-db6982769f37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:17:51.294349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.294206    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575decce-06ba-40bf-9e56-db6982769f37-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:51.299002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.295891    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca" pod "575decce-06ba-40bf-9e56-db6982769f37" (UID: "575decce-06ba-40bf-9e56-db6982769f37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:17:51.299002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.296079    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf" pod "575decce-06ba-40bf-9e56-db6982769f37" (UID: "575decce-06ba-40bf-9e56-db6982769f37"). InnerVolumeSpecName "kube-api-access-tfvcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:17:51.395580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.395519    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvcf\" (UniqueName: \"kubernetes.io/projected/575decce-06ba-40bf-9e56-db6982769f37-kube-api-access-tfvcf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:51.395580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.395577    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575decce-06ba-40bf-9e56-db6982769f37-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:17:51.901323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:51.901293    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:52.060933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:52.060903    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"
Nov 01 09:17:52.060933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:52.060924    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8" event={"ID":"575decce-06ba-40bf-9e56-db6982769f37","Type":"ContainerDied","Data":"4e0738fea2bcd14c119589723747849e7db716e99511ebc4bc1e57e743758f94"}
Nov 01 09:17:52.061147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:52.060972    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e0738fea2bcd14c119589723747849e7db716e99511ebc4bc1e57e743758f94"
Nov 01 09:17:52.087645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:52.087619    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:52.253974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:52.253942    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:52.675178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:52.675101    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:55.053133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:55.053101    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:55.217929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:55.217838    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:57.072180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:57.072123    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:17:58.892149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:17:58.892114    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:01.213708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.213671    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"]
Nov 01 09:18:01.214229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.214214    4314 cpu_manager.go:414] "containerMap: removing container" podUID="575decce-06ba-40bf-9e56-db6982769f37" containerName="revoke-keys"
Nov 01 09:18:01.214297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.214229    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575decce-06ba-40bf-9e56-db6982769f37" containerName="revoke-keys"
Nov 01 09:18:01.214354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.214342    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="575decce-06ba-40bf-9e56-db6982769f37" containerName="revoke-keys"
Nov 01 09:18:01.219240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.219222    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.271866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.271832    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"]
Nov 01 09:18:01.313195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.313158    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:18:01.313195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.313191    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:18:01.313428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.313197    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:18:01.314368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.314348    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:18:01.323162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.323136    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmnq\" (UniqueName: \"kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.323283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.323268    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.323336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.323323    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.424991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.424959    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.425178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.425068    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmnq\" (UniqueName: \"kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.425178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.425141    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.425756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.425736    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.427480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.427463    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.482762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.482736    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmnq\" (UniqueName: \"kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq\") pod \"maas-api-revoke-sub-6ee64da6-57cgj\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.533931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.533900    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:01.907349 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:01.907318    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc447073d_aeb9_4b4c_bd83_d569030fd9e4.slice/crio-c09275357d36a577027f245a872bd1c3099dff83a91a747bfb33a62bb806dfbf WatchSource:0}: Error finding container c09275357d36a577027f245a872bd1c3099dff83a91a747bfb33a62bb806dfbf: Status 404 returned error can't find the container with id c09275357d36a577027f245a872bd1c3099dff83a91a747bfb33a62bb806dfbf
Nov 01 09:18:01.907466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:01.907350    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"]
Nov 01 09:18:02.129031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:02.128992    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj" event={"ID":"c447073d-aeb9-4b4c-bd83-d569030fd9e4","Type":"ContainerStarted","Data":"c09275357d36a577027f245a872bd1c3099dff83a91a747bfb33a62bb806dfbf"}
Nov 01 09:18:03.147392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.147307    4314 generic.go:356] "Generic (PLEG): container finished" podID="c447073d-aeb9-4b4c-bd83-d569030fd9e4" containerID="5201486697d7910db864664448fea4ce5eb741e24daeee3216228e108f7d3796" exitCode=0
Nov 01 09:18:03.147787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.147393    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj" event={"ID":"c447073d-aeb9-4b4c-bd83-d569030fd9e4","Type":"ContainerDied","Data":"5201486697d7910db864664448fea4ce5eb741e24daeee3216228e108f7d3796"}
Nov 01 09:18:03.548992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.548951    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"]
Nov 01 09:18:03.553367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.553344    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.607251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.607221    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"]
Nov 01 09:18:03.654103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.654076    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.654270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.654126    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.654343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.654323    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8z8k\" (UniqueName: \"kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.690521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.690493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:03.755892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.755854    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8z8k\" (UniqueName: \"kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.756081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.755940    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.756081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.755981    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.756751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.756724    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.758335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.758314    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.813073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.813006    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8z8k\" (UniqueName: \"kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k\") pod \"maas-api-revoke-sub-1ff175c7-x9gd4\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:03.867446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:03.867416    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:04.235354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.235324    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"]
Nov 01 09:18:04.237296 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:04.237264    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767a0a5e_5d17_48d0_a141_70025df7e887.slice/crio-75e384918fb8da91d879bec0257562edc420b0634264bf7abfe2e25692a4bccf WatchSource:0}: Error finding container 75e384918fb8da91d879bec0257562edc420b0634264bf7abfe2e25692a4bccf: Status 404 returned error can't find the container with id 75e384918fb8da91d879bec0257562edc420b0634264bf7abfe2e25692a4bccf
Nov 01 09:18:04.278233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.278209    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:04.364147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.364114    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca\") pod \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") "
Nov 01 09:18:04.364374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.364166    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq\" (UniqueName: \"kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq\") pod \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") "
Nov 01 09:18:04.364374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.364259    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca\") pod \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\" (UID: \"c447073d-aeb9-4b4c-bd83-d569030fd9e4\") "
Nov 01 09:18:04.364615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.364585    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca" pod "c447073d-aeb9-4b4c-bd83-d569030fd9e4" (UID: "c447073d-aeb9-4b4c-bd83-d569030fd9e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:04.364824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.364809    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:04.366774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.366757    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca" pod "c447073d-aeb9-4b4c-bd83-d569030fd9e4" (UID: "c447073d-aeb9-4b4c-bd83-d569030fd9e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:04.366774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.366766    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq" pod "c447073d-aeb9-4b4c-bd83-d569030fd9e4" (UID: "c447073d-aeb9-4b4c-bd83-d569030fd9e4"). InnerVolumeSpecName "kube-api-access-wbmnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:04.466363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.466294    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbmnq\" (UniqueName: \"kubernetes.io/projected/c447073d-aeb9-4b4c-bd83-d569030fd9e4-kube-api-access-wbmnq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:04.466363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:04.466320    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c447073d-aeb9-4b4c-bd83-d569030fd9e4-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:05.162226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:05.162182    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4" event={"ID":"767a0a5e-5d17-48d0-a141-70025df7e887","Type":"ContainerStarted","Data":"c2e439b3ee754b22e8bc891a83a6dd9138b45f12f1b01fd98c094713ed2121c7"}
Nov 01 09:18:05.162387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:05.162264    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4" event={"ID":"767a0a5e-5d17-48d0-a141-70025df7e887","Type":"ContainerStarted","Data":"75e384918fb8da91d879bec0257562edc420b0634264bf7abfe2e25692a4bccf"}
Nov 01 09:18:05.164593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:05.163932    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"
Nov 01 09:18:05.164593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:05.163957    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj" event={"ID":"c447073d-aeb9-4b4c-bd83-d569030fd9e4","Type":"ContainerDied","Data":"c09275357d36a577027f245a872bd1c3099dff83a91a747bfb33a62bb806dfbf"}
Nov 01 09:18:05.164593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:05.163997    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09275357d36a577027f245a872bd1c3099dff83a91a747bfb33a62bb806dfbf"
Nov 01 09:18:05.272866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:05.272808    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4" podStartSLOduration=2.272793429 podStartE2EDuration="2.272793429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:05.270855233 +0000 UTC m=+2758.465249788" watchObservedRunningTime="2026-11-01 09:18:05.272793429 +0000 UTC m=+2758.467187984"
Nov 01 09:18:06.172354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:06.172298    4314 generic.go:356] "Generic (PLEG): container finished" podID="767a0a5e-5d17-48d0-a141-70025df7e887" containerID="c2e439b3ee754b22e8bc891a83a6dd9138b45f12f1b01fd98c094713ed2121c7" exitCode=0
Nov 01 09:18:06.172557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:06.172382    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4" event={"ID":"767a0a5e-5d17-48d0-a141-70025df7e887","Type":"ContainerDied","Data":"c2e439b3ee754b22e8bc891a83a6dd9138b45f12f1b01fd98c094713ed2121c7"}
Nov 01 09:18:06.228252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:06.228218    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:06.376080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:06.376049    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:06.945275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:06.945246    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:07.093549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.093506    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:07.327982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.327953    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:07.502630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.502516    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca\") pod \"767a0a5e-5d17-48d0-a141-70025df7e887\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") "
Nov 01 09:18:07.503092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.502679    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca\") pod \"767a0a5e-5d17-48d0-a141-70025df7e887\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") "
Nov 01 09:18:07.503092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.502784    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k\" (UniqueName: \"kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k\") pod \"767a0a5e-5d17-48d0-a141-70025df7e887\" (UID: \"767a0a5e-5d17-48d0-a141-70025df7e887\") "
Nov 01 09:18:07.503092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.502902    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca" pod "767a0a5e-5d17-48d0-a141-70025df7e887" (UID: "767a0a5e-5d17-48d0-a141-70025df7e887"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:07.503255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.503234    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:07.504668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.504642    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca" pod "767a0a5e-5d17-48d0-a141-70025df7e887" (UID: "767a0a5e-5d17-48d0-a141-70025df7e887"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:07.504810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.504775    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k" pod "767a0a5e-5d17-48d0-a141-70025df7e887" (UID: "767a0a5e-5d17-48d0-a141-70025df7e887"). InnerVolumeSpecName "kube-api-access-d8z8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:07.603953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.603924    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767a0a5e-5d17-48d0-a141-70025df7e887-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:07.603953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:07.603950    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8z8k\" (UniqueName: \"kubernetes.io/projected/767a0a5e-5d17-48d0-a141-70025df7e887-kube-api-access-d8z8k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:08.188548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:08.188508    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"
Nov 01 09:18:08.188740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:08.188555    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4" event={"ID":"767a0a5e-5d17-48d0-a141-70025df7e887","Type":"ContainerDied","Data":"75e384918fb8da91d879bec0257562edc420b0634264bf7abfe2e25692a4bccf"}
Nov 01 09:18:08.188740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:08.188610    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75e384918fb8da91d879bec0257562edc420b0634264bf7abfe2e25692a4bccf"
Nov 01 09:18:08.527300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:08.527270    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:08.684070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:08.684016    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:09.003601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:09.003572    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"]
Nov 01 09:18:09.003903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:09.003877    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy" containerID="cri-o://c690473df16882fad5abcb85a1843e468b49116816fcab8a5c2864fd27bd8fd7" gracePeriod=30
Nov 01 09:18:09.009518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:09.009483    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:11.323548 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:18:11.323501    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:18:14.270559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.268152    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:14.274551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.273906    4314 generic.go:356] "Generic (PLEG): container finished" podID="16c92efb-3798-491a-ac10-165626a4f87f" containerID="c690473df16882fad5abcb85a1843e468b49116816fcab8a5c2864fd27bd8fd7" exitCode=0
Nov 01 09:18:14.274551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.274046    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" event={"ID":"16c92efb-3798-491a-ac10-165626a4f87f","Type":"ContainerDied","Data":"c690473df16882fad5abcb85a1843e468b49116816fcab8a5c2864fd27bd8fd7"}
Nov 01 09:18:14.274551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.274106    4314 scope.go:122] "RemoveContainer" containerID="99e471e65f07de1b92cdabc5e02cdc5360e9763c76bcb39dd783af68855ce64e"
Nov 01 09:18:14.623185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.623141    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:14.795679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.795626    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:14.882862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:14.882750    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:15.403951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.403913    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:18:15.443376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443335    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443448    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443470    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443521    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443562    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443642    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443693    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.443900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443764    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:18:15.443900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.443807    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.444434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.444407    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:18:15.444674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.444427    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:18:15.444674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.444574    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:18:15.444897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.444875    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.445043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.445027    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.445172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.445158    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.445276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.445264    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.445930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.445893    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:15.446184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.446154    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:18:15.446279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.446199    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:18:15.446927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.446911    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "kube-api-access-4kzrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:15.546519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.546478    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert\") pod \"16c92efb-3798-491a-ac10-165626a4f87f\" (UID: \"16c92efb-3798-491a-ac10-165626a4f87f\") "
Nov 01 09:18:15.546888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.546861    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert" pod "16c92efb-3798-491a-ac10-165626a4f87f" (UID: "16c92efb-3798-491a-ac10-165626a4f87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:15.547030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.547013    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c92efb-3798-491a-ac10-165626a4f87f-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.547086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.547034    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c92efb-3798-491a-ac10-165626a4f87f-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.547086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.547043    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.547086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.547053    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kzrl\" (UniqueName: \"kubernetes.io/projected/16c92efb-3798-491a-ac10-165626a4f87f-kube-api-access-4kzrl\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.547086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.547063    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c92efb-3798-491a-ac10-165626a4f87f-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:15.724439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:15.724337    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:16.354256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.354222    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5" event={"ID":"16c92efb-3798-491a-ac10-165626a4f87f","Type":"ContainerDied","Data":"95ada2f3e03d70b85643312ec3699efc0384f99e6a669cfac273b3903f4c818a"}
Nov 01 09:18:16.354449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.354277    4314 scope.go:122] "RemoveContainer" containerID="c690473df16882fad5abcb85a1843e468b49116816fcab8a5c2864fd27bd8fd7"
Nov 01 09:18:16.354449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.354232    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"
Nov 01 09:18:16.556627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.556593    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"]
Nov 01 09:18:16.602462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.602423    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1c173-openshift-default-56fdcc8c97-n2bd5"]
Nov 01 09:18:16.880512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.880475    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"]
Nov 01 09:18:16.881059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881042    4314 cpu_manager.go:414] "containerMap: removing container" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy"
Nov 01 09:18:16.881059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881060    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy"
Nov 01 09:18:16.881187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881069    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c447073d-aeb9-4b4c-bd83-d569030fd9e4" containerName="revoke-keys"
Nov 01 09:18:16.881187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881076    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c447073d-aeb9-4b4c-bd83-d569030fd9e4" containerName="revoke-keys"
Nov 01 09:18:16.881187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881102    4314 cpu_manager.go:414] "containerMap: removing container" podUID="767a0a5e-5d17-48d0-a141-70025df7e887" containerName="revoke-keys"
Nov 01 09:18:16.881187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881108    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767a0a5e-5d17-48d0-a141-70025df7e887" containerName="revoke-keys"
Nov 01 09:18:16.881308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881188    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy"
Nov 01 09:18:16.881308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881196    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c447073d-aeb9-4b4c-bd83-d569030fd9e4" containerName="revoke-keys"
Nov 01 09:18:16.881308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881206    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy"
Nov 01 09:18:16.881308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.881213    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="767a0a5e-5d17-48d0-a141-70025df7e887" containerName="revoke-keys"
Nov 01 09:18:16.886388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.886370    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:16.929816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.929792    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c92efb-3798-491a-ac10-165626a4f87f" path="/var/lib/kubelet/pods/16c92efb-3798-491a-ac10-165626a4f87f/volumes"
Nov 01 09:18:16.935244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.935217    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:18:16.935396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.935263    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:18:16.935464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.935412    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:18:16.935464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.935421    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:18:16.939619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.939595    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"]
Nov 01 09:18:16.962891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.962861    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5x92\" (UniqueName: \"kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:16.963042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.962912    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:16.963113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:16.963066    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.064477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.064432    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5x92\" (UniqueName: \"kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.064688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.064501    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.064688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.064632    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.065311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.065289    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.067045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.067017    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.126894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.126863    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5x92\" (UniqueName: \"kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92\") pod \"maas-api-revoke-sub-d69b7da9-4t5bp\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.199744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.199668    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:17.572462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:17.572432    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"]
Nov 01 09:18:17.573717 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:17.573682    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984feac5_a47f_4a8a_b5eb_cfd7f3ae6d3f.slice/crio-1123037f113829de4cdacd637eb9905cbb6bf319a7da66cff3b91ef14385bea0 WatchSource:0}: Error finding container 1123037f113829de4cdacd637eb9905cbb6bf319a7da66cff3b91ef14385bea0: Status 404 returned error can't find the container with id 1123037f113829de4cdacd637eb9905cbb6bf319a7da66cff3b91ef14385bea0
Nov 01 09:18:18.371437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:18.371399    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp" event={"ID":"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f","Type":"ContainerStarted","Data":"29cf1f04664c508684469fbf1148fd5ad00f4c74c6befb96e5a163a0a2c9b862"}
Nov 01 09:18:18.371642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:18.371463    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp" event={"ID":"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f","Type":"ContainerStarted","Data":"1123037f113829de4cdacd637eb9905cbb6bf319a7da66cff3b91ef14385bea0"}
Nov 01 09:18:18.482835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:18.482778    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp" podStartSLOduration=2.482763261 podStartE2EDuration="2.482763261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:18.479969345 +0000 UTC m=+2771.674363891" watchObservedRunningTime="2026-11-01 09:18:18.482763261 +0000 UTC m=+2771.677157814"
Nov 01 09:18:19.210389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:19.210353    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:19.385879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:19.385842    4314 generic.go:356] "Generic (PLEG): container finished" podID="984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" containerID="29cf1f04664c508684469fbf1148fd5ad00f4c74c6befb96e5a163a0a2c9b862" exitCode=0
Nov 01 09:18:19.386067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:19.385916    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp" event={"ID":"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f","Type":"ContainerDied","Data":"29cf1f04664c508684469fbf1148fd5ad00f4c74c6befb96e5a163a0a2c9b862"}
Nov 01 09:18:20.529684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.529652    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:20.608334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.608277    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92\" (UniqueName: \"kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92\") pod \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") "
Nov 01 09:18:20.608578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.608557    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca\") pod \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") "
Nov 01 09:18:20.608687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.608670    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca\") pod \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\" (UID: \"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f\") "
Nov 01 09:18:20.608875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.608848    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca" pod "984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" (UID: "984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:20.609110 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.609094    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:20.610556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.610519    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92" pod "984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" (UID: "984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f"). InnerVolumeSpecName "kube-api-access-v5x92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:20.610655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.610630    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca" pod "984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" (UID: "984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:20.710782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.710693    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:20.710782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:20.710728    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5x92\" (UniqueName: \"kubernetes.io/projected/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f-kube-api-access-v5x92\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:21.400783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:21.400748    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp" event={"ID":"984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f","Type":"ContainerDied","Data":"1123037f113829de4cdacd637eb9905cbb6bf319a7da66cff3b91ef14385bea0"}
Nov 01 09:18:21.400967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:21.400797    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1123037f113829de4cdacd637eb9905cbb6bf319a7da66cff3b91ef14385bea0"
Nov 01 09:18:21.400967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:21.400813    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"
Nov 01 09:18:22.853994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.853958    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"]
Nov 01 09:18:22.854503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.854490    4314 cpu_manager.go:414] "containerMap: removing container" podUID="984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" containerName="revoke-keys"
Nov 01 09:18:22.854565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.854504    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" containerName="revoke-keys"
Nov 01 09:18:22.854599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.854565    4314 cpu_manager.go:414] "containerMap: removing container" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy"
Nov 01 09:18:22.854599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.854571    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c92efb-3798-491a-ac10-165626a4f87f" containerName="istio-proxy"
Nov 01 09:18:22.854688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.854678    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" containerName="revoke-keys"
Nov 01 09:18:22.858166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.858150    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.934861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.934826    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nns6\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.934941    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935014    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935054    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935103    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935187    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935338    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935387    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.935464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.935450    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:22.944716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:22.944689    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"]
Nov 01 09:18:23.036688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036650    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.036688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036688    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.036952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036713    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.036952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036757    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nns6\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.036952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036800    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.036952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036866    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.036952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.036913    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037023    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037072    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037128    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037159    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037220    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037431    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.037985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.037958    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.039267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.039249    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.039408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.039393    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.098749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.098721    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.098897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.098793    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nns6\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.171480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.171409    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:23.590298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.590269    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"]
Nov 01 09:18:23.590743 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:23.590717    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51883dec_a44f_4fa9_9032_b831d3551b60.slice/crio-1decddf4cd1d7d83f74de0f1423c1f8eb1903ccf22b88afe60934d93831c7dc5 WatchSource:0}: Error finding container 1decddf4cd1d7d83f74de0f1423c1f8eb1903ccf22b88afe60934d93831c7dc5: Status 404 returned error can't find the container with id 1decddf4cd1d7d83f74de0f1423c1f8eb1903ccf22b88afe60934d93831c7dc5
Nov 01 09:18:23.593214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.593179    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:23.593292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.593248    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:23.593292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:23.593279    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:24.189336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.189293    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:24.323081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.323045    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"]
Nov 01 09:18:24.327239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.327223    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.383555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.382990    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:18:24.385349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.383750    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:18:24.385349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.383820    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:18:24.385349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.383908    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:18:24.385349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.385064    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"]
Nov 01 09:18:24.423992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.423919    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" event={"ID":"51883dec-a44f-4fa9-9032-b831d3551b60","Type":"ContainerStarted","Data":"8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b"}
Nov 01 09:18:24.423992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.423981    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" event={"ID":"51883dec-a44f-4fa9-9032-b831d3551b60","Type":"ContainerStarted","Data":"1decddf4cd1d7d83f74de0f1423c1f8eb1903ccf22b88afe60934d93831c7dc5"}
Nov 01 09:18:24.455328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.455258    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.455328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.455307    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8p4\" (UniqueName: \"kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.455501 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.455399    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.557115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.557072    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.557322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.557141    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8p4\" (UniqueName: \"kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.557322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.557187    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.557946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.557916    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.566564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.563987    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.585492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.584868    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podStartSLOduration=2.584846942 podStartE2EDuration="2.584846942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:24.553757095 +0000 UTC m=+2777.748151661" watchObservedRunningTime="2026-11-01 09:18:24.584846942 +0000 UTC m=+2777.779241499"
Nov 01 09:18:24.616769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.616738    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8p4\" (UniqueName: \"kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4\") pod \"maas-api-revoke-sub-67e8be07-b9tl4\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:24.641307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:24.641274    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:25.040220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.040182    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"]
Nov 01 09:18:25.041095 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:25.041050    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a752e6e_ea64_459e_adc8_6a32e82a73be.slice/crio-d195eb068fa0716decb6432afef65a36873121b6e4eb286d3d69727583c0648d WatchSource:0}: Error finding container d195eb068fa0716decb6432afef65a36873121b6e4eb286d3d69727583c0648d: Status 404 returned error can't find the container with id d195eb068fa0716decb6432afef65a36873121b6e4eb286d3d69727583c0648d
Nov 01 09:18:25.183559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.178564    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:25.183559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.178754    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 09:18:25.183559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.178802    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 09:18:25.432396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.432274    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4" event={"ID":"4a752e6e-ea64-459e-adc8-6a32e82a73be","Type":"ContainerStarted","Data":"d195eb068fa0716decb6432afef65a36873121b6e4eb286d3d69727583c0648d"}
Nov 01 09:18:25.832280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.832229    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"]
Nov 01 09:18:25.836655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.836626    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:25.890479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:25.890446    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"]
Nov 01 09:18:26.002730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.002695    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.002936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.002884    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pwk\" (UniqueName: \"kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.003086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.003065    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.104181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.104092    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.104391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.104196    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pwk\" (UniqueName: \"kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.104391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.104271    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.104906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.104876    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.106736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.106712    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.184159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.184124    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 09:18:26.184339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.184201    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 09:18:26.185602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.185579    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pwk\" (UniqueName: \"kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk\") pod \"maas-api-revoke-sub-fa3b093c-f4q7c\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.441122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.441044    4314 generic.go:356] "Generic (PLEG): container finished" podID="4a752e6e-ea64-459e-adc8-6a32e82a73be" containerID="8a0dd27d8290507dab89c4f251286fca5fcadb8ca1b07b6b5beeeacf39d5738b" exitCode=0
Nov 01 09:18:26.441523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.441127    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4" event={"ID":"4a752e6e-ea64-459e-adc8-6a32e82a73be","Type":"ContainerDied","Data":"8a0dd27d8290507dab89c4f251286fca5fcadb8ca1b07b6b5beeeacf39d5738b"}
Nov 01 09:18:26.451699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.451678    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:26.846202 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:26.846172    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa71b344_3b51_4a78_8c55_fb9a28829eac.slice/crio-147ba513b4b55c479ca8ed8e38f058ff69882c30818d163cfd383679f457a6fc WatchSource:0}: Error finding container 147ba513b4b55c479ca8ed8e38f058ff69882c30818d163cfd383679f457a6fc: Status 404 returned error can't find the container with id 147ba513b4b55c479ca8ed8e38f058ff69882c30818d163cfd383679f457a6fc
Nov 01 09:18:26.846334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:26.846311    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"]
Nov 01 09:18:27.180556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.178403    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 09:18:27.180556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.178489    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 09:18:27.449587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.449551    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c" event={"ID":"aa71b344-3b51-4a78-8c55-fb9a28829eac","Type":"ContainerStarted","Data":"be681ab71d9762d3875ed171a54b0c5082885cb35af06a4cb6fc54e3b0436d31"}
Nov 01 09:18:27.449994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.449609    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c" event={"ID":"aa71b344-3b51-4a78-8c55-fb9a28829eac","Type":"ContainerStarted","Data":"147ba513b4b55c479ca8ed8e38f058ff69882c30818d163cfd383679f457a6fc"}
Nov 01 09:18:27.636747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.636720    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:27.724587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.724481    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca\") pod \"4a752e6e-ea64-459e-adc8-6a32e82a73be\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") "
Nov 01 09:18:27.724759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.724657    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4\" (UniqueName: \"kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4\") pod \"4a752e6e-ea64-459e-adc8-6a32e82a73be\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") "
Nov 01 09:18:27.724759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.724708    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca\") pod \"4a752e6e-ea64-459e-adc8-6a32e82a73be\" (UID: \"4a752e6e-ea64-459e-adc8-6a32e82a73be\") "
Nov 01 09:18:27.724874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.724805    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca" pod "4a752e6e-ea64-459e-adc8-6a32e82a73be" (UID: "4a752e6e-ea64-459e-adc8-6a32e82a73be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:27.725249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.725225    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:27.726943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.726914    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4" pod "4a752e6e-ea64-459e-adc8-6a32e82a73be" (UID: "4a752e6e-ea64-459e-adc8-6a32e82a73be"). InnerVolumeSpecName "kube-api-access-kt8p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:27.727038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.726977    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca" pod "4a752e6e-ea64-459e-adc8-6a32e82a73be" (UID: "4a752e6e-ea64-459e-adc8-6a32e82a73be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:27.749095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.749046    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c" podStartSLOduration=2.749033302 podStartE2EDuration="2.749033302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:27.57977738 +0000 UTC m=+2780.774171935" watchObservedRunningTime="2026-11-01 09:18:27.749033302 +0000 UTC m=+2780.943427857"
Nov 01 09:18:27.826662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.826623    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8p4\" (UniqueName: \"kubernetes.io/projected/4a752e6e-ea64-459e-adc8-6a32e82a73be-kube-api-access-kt8p4\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:27.826662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:27.826658    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a752e6e-ea64-459e-adc8-6a32e82a73be-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:28.178557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.174583    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 09:18:28.178557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.174648    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 09:18:28.458581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.457962    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4" event={"ID":"4a752e6e-ea64-459e-adc8-6a32e82a73be","Type":"ContainerDied","Data":"d195eb068fa0716decb6432afef65a36873121b6e4eb286d3d69727583c0648d"}
Nov 01 09:18:28.458581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.458033    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d195eb068fa0716decb6432afef65a36873121b6e4eb286d3d69727583c0648d"
Nov 01 09:18:28.458581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.458115    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"
Nov 01 09:18:28.485649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.485611    4314 generic.go:356] "Generic (PLEG): container finished" podID="aa71b344-3b51-4a78-8c55-fb9a28829eac" containerID="be681ab71d9762d3875ed171a54b0c5082885cb35af06a4cb6fc54e3b0436d31" exitCode=0
Nov 01 09:18:28.485840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:28.485693    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c" event={"ID":"aa71b344-3b51-4a78-8c55-fb9a28829eac","Type":"ContainerDied","Data":"be681ab71d9762d3875ed171a54b0c5082885cb35af06a4cb6fc54e3b0436d31"}
Nov 01 09:18:29.108196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.108158    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:29.179559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.175936    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 09:18:29.179559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.176003    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 09:18:29.652236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.652210    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:29.749631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.749524    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk\" (UniqueName: \"kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk\") pod \"aa71b344-3b51-4a78-8c55-fb9a28829eac\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") "
Nov 01 09:18:29.749791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.749662    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca\") pod \"aa71b344-3b51-4a78-8c55-fb9a28829eac\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") "
Nov 01 09:18:29.749791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.749694    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca\") pod \"aa71b344-3b51-4a78-8c55-fb9a28829eac\" (UID: \"aa71b344-3b51-4a78-8c55-fb9a28829eac\") "
Nov 01 09:18:29.750137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.750100    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca" pod "aa71b344-3b51-4a78-8c55-fb9a28829eac" (UID: "aa71b344-3b51-4a78-8c55-fb9a28829eac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:29.750283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.750237    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:29.751849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.751815    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca" pod "aa71b344-3b51-4a78-8c55-fb9a28829eac" (UID: "aa71b344-3b51-4a78-8c55-fb9a28829eac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:29.752257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.752232    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk" pod "aa71b344-3b51-4a78-8c55-fb9a28829eac" (UID: "aa71b344-3b51-4a78-8c55-fb9a28829eac"). InnerVolumeSpecName "kube-api-access-k2pwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:29.852185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.852142    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pwk\" (UniqueName: \"kubernetes.io/projected/aa71b344-3b51-4a78-8c55-fb9a28829eac-kube-api-access-k2pwk\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:29.852185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:29.852189    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71b344-3b51-4a78-8c55-fb9a28829eac-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:30.172013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:30.171927    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 09:18:30.172013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:30.171988    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 09:18:30.502207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:30.502172    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"
Nov 01 09:18:30.502397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:30.502204    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c" event={"ID":"aa71b344-3b51-4a78-8c55-fb9a28829eac","Type":"ContainerDied","Data":"147ba513b4b55c479ca8ed8e38f058ff69882c30818d163cfd383679f457a6fc"}
Nov 01 09:18:30.502397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:30.502256    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147ba513b4b55c479ca8ed8e38f058ff69882c30818d163cfd383679f457a6fc"
Nov 01 09:18:31.591839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:31.591799    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:31.592782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:31.592740    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:31.593673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:31.593649    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:18:32.413209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413176    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"]
Nov 01 09:18:32.413750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413736    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a752e6e-ea64-459e-adc8-6a32e82a73be" containerName="revoke-keys"
Nov 01 09:18:32.413820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413751    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a752e6e-ea64-459e-adc8-6a32e82a73be" containerName="revoke-keys"
Nov 01 09:18:32.413820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413791    4314 cpu_manager.go:414] "containerMap: removing container" podUID="aa71b344-3b51-4a78-8c55-fb9a28829eac" containerName="revoke-keys"
Nov 01 09:18:32.413820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413796    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa71b344-3b51-4a78-8c55-fb9a28829eac" containerName="revoke-keys"
Nov 01 09:18:32.413920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413878    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa71b344-3b51-4a78-8c55-fb9a28829eac" containerName="revoke-keys"
Nov 01 09:18:32.413920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.413890    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a752e6e-ea64-459e-adc8-6a32e82a73be" containerName="revoke-keys"
Nov 01 09:18:32.417674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.417657    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.466404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.466376    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:18:32.466756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.466703    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:18:32.472151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.472122    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"]
Nov 01 09:18:32.552256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.552224    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:18:32.552407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.552345    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:18:32.587470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.587440    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2hf\" (UniqueName: \"kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.587658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.587490    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.587658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.587580    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.688919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.688830    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2hf\" (UniqueName: \"kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.689407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.688929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.689407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.689115    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.689784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.689762    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.691338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.691318    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:32.746089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:32.746058    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2hf\" (UniqueName: \"kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf\") pod \"maas-api-revoke-sub-2ecd3c9f-bkgsc\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:33.033749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:33.033712    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:33.403734 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:33.403687    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ad0cd3_ba79_49ed_8b6c_041ab8b3aba1.slice/crio-95c30c5e5ddc1aa9c0ab092d2db589d9fe6cead9ab5bed7b6ec8abad4487e800 WatchSource:0}: Error finding container 95c30c5e5ddc1aa9c0ab092d2db589d9fe6cead9ab5bed7b6ec8abad4487e800: Status 404 returned error can't find the container with id 95c30c5e5ddc1aa9c0ab092d2db589d9fe6cead9ab5bed7b6ec8abad4487e800
Nov 01 09:18:33.403985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:33.403963    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"]
Nov 01 09:18:33.525389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:33.525354    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc" event={"ID":"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1","Type":"ContainerStarted","Data":"95c30c5e5ddc1aa9c0ab092d2db589d9fe6cead9ab5bed7b6ec8abad4487e800"}
Nov 01 09:18:33.990512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:33.990469    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:34.533372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:34.533287    4314 generic.go:356] "Generic (PLEG): container finished" podID="48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" containerID="c4998c199fea58b9c7dd91892263724d3e2313569a10e402000d437088126b8e" exitCode=0
Nov 01 09:18:34.533558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:34.533369    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc" event={"ID":"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1","Type":"ContainerDied","Data":"c4998c199fea58b9c7dd91892263724d3e2313569a10e402000d437088126b8e"}
Nov 01 09:18:35.083502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.083464    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"]
Nov 01 09:18:35.087875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.087857    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.141770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.141737    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"]
Nov 01 09:18:35.219908 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.219871    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.220084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.220064    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-868dr\" (UniqueName: \"kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.220171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.220157    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.321202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.321168    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-868dr\" (UniqueName: \"kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.321407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.321229    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.321407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.321300    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.321865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.321841    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.323647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.323623    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.378463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.378393    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-868dr\" (UniqueName: \"kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr\") pod \"maas-api-revoke-sub-0b0fce89-scwgx\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.403018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.402991    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:35.681459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.681432    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:35.777216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.777192    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"]
Nov 01 09:18:35.777702 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:35.777682    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175abac6_0887_435d_a53f_929b2cafa367.slice/crio-0bd1b89567d8a43bc2dccd360ad8836a1c44c326b29055acd98de031a6e87a1b WatchSource:0}: Error finding container 0bd1b89567d8a43bc2dccd360ad8836a1c44c326b29055acd98de031a6e87a1b: Status 404 returned error can't find the container with id 0bd1b89567d8a43bc2dccd360ad8836a1c44c326b29055acd98de031a6e87a1b
Nov 01 09:18:35.827952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.827917    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf\" (UniqueName: \"kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf\") pod \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") "
Nov 01 09:18:35.828108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.828018    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca\") pod \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") "
Nov 01 09:18:35.828195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.828175    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca\") pod \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\" (UID: \"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1\") "
Nov 01 09:18:35.828810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.828776    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca" pod "48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" (UID: "48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:35.830146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.830116    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf" pod "48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" (UID: "48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1"). InnerVolumeSpecName "kube-api-access-fs2hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:35.830239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.830198    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca" pod "48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" (UID: "48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:35.929521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.929425    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2hf\" (UniqueName: \"kubernetes.io/projected/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-kube-api-access-fs2hf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:35.929521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.929455    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:35.929521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:35.929465    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:36.549732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:36.549696    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"
Nov 01 09:18:36.549732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:36.549720    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc" event={"ID":"48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1","Type":"ContainerDied","Data":"95c30c5e5ddc1aa9c0ab092d2db589d9fe6cead9ab5bed7b6ec8abad4487e800"}
Nov 01 09:18:36.550202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:36.549772    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c30c5e5ddc1aa9c0ab092d2db589d9fe6cead9ab5bed7b6ec8abad4487e800"
Nov 01 09:18:36.551429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:36.551396    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx" event={"ID":"175abac6-0887-435d-a53f-929b2cafa367","Type":"ContainerStarted","Data":"30f6a1e284d884b5d26ba2e4513452e2a00c80d1ba66c79bc44af57ea1f731cc"}
Nov 01 09:18:36.551612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:36.551443    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx" event={"ID":"175abac6-0887-435d-a53f-929b2cafa367","Type":"ContainerStarted","Data":"0bd1b89567d8a43bc2dccd360ad8836a1c44c326b29055acd98de031a6e87a1b"}
Nov 01 09:18:36.661225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:36.661177    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx" podStartSLOduration=1.661159213 podStartE2EDuration="1.661159213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:36.65879913 +0000 UTC m=+2789.853193686" watchObservedRunningTime="2026-11-01 09:18:36.661159213 +0000 UTC m=+2789.855553772"
Nov 01 09:18:37.562657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:37.562622    4314 generic.go:356] "Generic (PLEG): container finished" podID="175abac6-0887-435d-a53f-929b2cafa367" containerID="30f6a1e284d884b5d26ba2e4513452e2a00c80d1ba66c79bc44af57ea1f731cc" exitCode=0
Nov 01 09:18:37.563088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:37.562708    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx" event={"ID":"175abac6-0887-435d-a53f-929b2cafa367","Type":"ContainerDied","Data":"30f6a1e284d884b5d26ba2e4513452e2a00c80d1ba66c79bc44af57ea1f731cc"}
Nov 01 09:18:38.002031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.001993    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"]
Nov 01 09:18:38.002681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.002658    4314 cpu_manager.go:414] "containerMap: removing container" podUID="48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" containerName="revoke-keys"
Nov 01 09:18:38.002681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.002677    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" containerName="revoke-keys"
Nov 01 09:18:38.002855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.002799    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" containerName="revoke-keys"
Nov 01 09:18:38.006286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.006271    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.106798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.106758    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"]
Nov 01 09:18:38.158909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.158866    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.158925    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159036    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159079    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159132    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159218    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159240    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159267    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w54w\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.159348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.159335    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.260966    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261010    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261048    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w54w\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261097    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261204    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261249    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261291    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261332    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261373    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261440    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261577    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261652    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261744    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.261949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.261929    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.263861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.263840    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.263942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.263893    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.334495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.334460    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.334695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.334470    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w54w\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.619950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.619912    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:38.740748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.740715    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:38.747609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.747309    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:38.775808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.775084    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr\" (UniqueName: \"kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr\") pod \"175abac6-0887-435d-a53f-929b2cafa367\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") "
Nov 01 09:18:38.775808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.775310    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca\") pod \"175abac6-0887-435d-a53f-929b2cafa367\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") "
Nov 01 09:18:38.775808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.775441    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca\") pod \"175abac6-0887-435d-a53f-929b2cafa367\" (UID: \"175abac6-0887-435d-a53f-929b2cafa367\") "
Nov 01 09:18:38.776222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.776090    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca" pod "175abac6-0887-435d-a53f-929b2cafa367" (UID: "175abac6-0887-435d-a53f-929b2cafa367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:38.777635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.777605    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr" pod "175abac6-0887-435d-a53f-929b2cafa367" (UID: "175abac6-0887-435d-a53f-929b2cafa367"). InnerVolumeSpecName "kube-api-access-868dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:38.778156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.778130    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca" pod "175abac6-0887-435d-a53f-929b2cafa367" (UID: "175abac6-0887-435d-a53f-929b2cafa367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:38.877019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.876947    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175abac6-0887-435d-a53f-929b2cafa367-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:38.877019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.876974    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175abac6-0887-435d-a53f-929b2cafa367-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:38.877019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.876984    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-868dr\" (UniqueName: \"kubernetes.io/projected/175abac6-0887-435d-a53f-929b2cafa367-kube-api-access-868dr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:38.899633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:38.899579    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:39.046479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.046447    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:39.073372 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:39.073342    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7b8005_f393_48e3_91f1_0d01bdc12bb9.slice/crio-b7f433096c6a647c3a7aefce261ef9f7f9f44c17f05e893764edc2c320f6557a WatchSource:0}: Error finding container b7f433096c6a647c3a7aefce261ef9f7f9f44c17f05e893764edc2c320f6557a: Status 404 returned error can't find the container with id b7f433096c6a647c3a7aefce261ef9f7f9f44c17f05e893764edc2c320f6557a
Nov 01 09:18:39.073482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.073377    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"]
Nov 01 09:18:39.075938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.075897    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:39.076060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.075979    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:39.076060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.076015    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:39.579456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.579421    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"
Nov 01 09:18:39.579456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.579447    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx" event={"ID":"175abac6-0887-435d-a53f-929b2cafa367","Type":"ContainerDied","Data":"0bd1b89567d8a43bc2dccd360ad8836a1c44c326b29055acd98de031a6e87a1b"}
Nov 01 09:18:39.579766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.579504    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd1b89567d8a43bc2dccd360ad8836a1c44c326b29055acd98de031a6e87a1b"
Nov 01 09:18:39.581383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.581359    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" event={"ID":"5b7b8005-f393-48e3-91f1-0d01bdc12bb9","Type":"ContainerStarted","Data":"ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab"}
Nov 01 09:18:39.581573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.581397    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" event={"ID":"5b7b8005-f393-48e3-91f1-0d01bdc12bb9","Type":"ContainerStarted","Data":"b7f433096c6a647c3a7aefce261ef9f7f9f44c17f05e893764edc2c320f6557a"}
Nov 01 09:18:39.621138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.621104    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:39.697140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:39.697071    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podStartSLOduration=2.69705436 podStartE2EDuration="2.69705436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:39.694702757 +0000 UTC m=+2792.889097315" watchObservedRunningTime="2026-11-01 09:18:39.69705436 +0000 UTC m=+2792.891448917"
Nov 01 09:18:40.622460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:40.622422    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:40.622911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:40.622484    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:41.324065 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:18:41.324028    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:18:41.620949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:41.620851    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:41.620949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:41.620913    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:42.620752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:42.620707    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:42.621293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:42.620773    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:43.629559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:43.628508    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:43.629559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:43.628596    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:44.618712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.618660    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.619653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.619029    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.620707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.620599    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:44.620707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.620657    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:44.683875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.683829    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.789677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.789622    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.845558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.845169    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.925615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:44.925147    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:45.033698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.033649    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.106161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.106057    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:45.212618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.211690    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.581966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.572236    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.639600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.637001    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.639600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.637479    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:18:45.643553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.641001    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.694577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.669524    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:18:45.721592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.695913    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.721592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.696343    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:18:45.780579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.772332    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:46.079619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:45.806606    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.173714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.040910    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 09:18:46.184985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.184945    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:18:46.185323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.185303    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:18:46.185891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.052057    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 09:18:46.188854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.188791    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:18:46.215463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.198224    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:46.215463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.198291    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:46.350478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.350314    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:18:46.395559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.393809    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"]
Nov 01 09:18:46.415558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.407854    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.429336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.427886    4314 cpu_manager.go:414] "containerMap: removing container" podUID="175abac6-0887-435d-a53f-929b2cafa367" containerName="revoke-keys"
Nov 01 09:18:46.429611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.429590    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175abac6-0887-435d-a53f-929b2cafa367" containerName="revoke-keys"
Nov 01 09:18:46.429935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.429920    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="175abac6-0887-435d-a53f-929b2cafa367" containerName="revoke-keys"
Nov 01 09:18:46.441567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.441052    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:46.450690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.450659    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:18:46.451003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.450989    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:18:46.455215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.455193    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:18:46.455892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.455872    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"]
Nov 01 09:18:46.456051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.451485    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.457937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.457913    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:18:46.513764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.488890    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:18:46.517160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.517131    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-035625-openshift-default-dockercfg-vl4pq\""
Nov 01 09:18:46.527414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.527383    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.547339    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.547402    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.547642    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.547683    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.548517    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds26h\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.548708    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.548773    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.548829    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.551635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.548863    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.556781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.556754    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:18:46.622117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.622033    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:46.622117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.622092    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:46.652186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652141    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652201    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652265    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds26h\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652361    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652405    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652444    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652484    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652620    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.652802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.652656    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.653240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.653211    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.653936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.653908    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.654190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.654170    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.657229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.657200    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.657490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.657467    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.658841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.658815    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.661460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.661433    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.711648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.711613    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds26h\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.715034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.715006    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token\") pod \"e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") " pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:46.829747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:46.829712    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:47.263573 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:47.263527    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73eaa51_7bee_44b5_b178_c774c7984717.slice/crio-c8b6f8055726748cd8c3daed863c55138946160033bc50ae428a7e061744da69 WatchSource:0}: Error finding container c8b6f8055726748cd8c3daed863c55138946160033bc50ae428a7e061744da69: Status 404 returned error can't find the container with id c8b6f8055726748cd8c3daed863c55138946160033bc50ae428a7e061744da69
Nov 01 09:18:47.263824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.263795    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"]
Nov 01 09:18:47.266287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.266244    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:47.266390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.266330    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:47.266390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.266373    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:47.462804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.460164    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" event={"ID":"d73eaa51-7bee-44b5-b178-c774c7984717","Type":"ContainerStarted","Data":"32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"}
Nov 01 09:18:47.462804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.460220    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" event={"ID":"d73eaa51-7bee-44b5-b178-c774c7984717","Type":"ContainerStarted","Data":"c8b6f8055726748cd8c3daed863c55138946160033bc50ae428a7e061744da69"}
Nov 01 09:18:47.621210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.621117    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 01 09:18:47.621210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.621174    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 01 09:18:47.651186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.651120    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podStartSLOduration=2.651100871 podStartE2EDuration="2.651100871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:47.644460124 +0000 UTC m=+2800.838854678" watchObservedRunningTime="2026-11-01 09:18:47.651100871 +0000 UTC m=+2800.845495427"
Nov 01 09:18:47.830315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:47.830282    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:48.176077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.176040    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:48.684415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.684367    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"]
Nov 01 09:18:48.688419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.688397    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.755561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.755093    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a87a1d-openshift-default-dockercfg-jwv8b\""
Nov 01 09:18:48.785209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.785158    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:48.808566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.808506    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"]
Nov 01 09:18:48.809187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809150    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809229    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm49z\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809259    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809293    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809337    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809367    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809399    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809459    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.809762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.809484    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.844889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.844855    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:48.910219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910180    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910233    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910291    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910322    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910406    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910483    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm49z\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910518    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910573    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910647    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910659    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910733    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.910998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.910976    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.911266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.911245    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.911438 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.911411    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.913672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.913640    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.913791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.913764    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.971876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.971801    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:48.976725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:48.976696    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm49z\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:49.003909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.003873    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:49.128276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.128215    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:49.431899 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:49.431868    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07cb848_0d0f_4f34_8b25_49a143dee3d1.slice/crio-13e4254142824e54d3994f6e7b9ca19450e79c8170c0d0fc689beeea0a355401 WatchSource:0}: Error finding container 13e4254142824e54d3994f6e7b9ca19450e79c8170c0d0fc689beeea0a355401: Status 404 returned error can't find the container with id 13e4254142824e54d3994f6e7b9ca19450e79c8170c0d0fc689beeea0a355401
Nov 01 09:18:49.432275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.432243    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"]
Nov 01 09:18:49.434716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.434680    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:49.434811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.434746    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:49.434811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.434775    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:18:49.486175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.486086    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" event={"ID":"c07cb848-0d0f-4f34-8b25-49a143dee3d1","Type":"ContainerStarted","Data":"13e4254142824e54d3994f6e7b9ca19450e79c8170c0d0fc689beeea0a355401"}
Nov 01 09:18:49.487550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.487138    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:49.489311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.488462    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:18:49.666708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.666668    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:18:49.666945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:49.666762    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:50.323906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.323854    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:50.325373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.325347    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:50.326374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.326349    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:18:50.494989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.494953    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" event={"ID":"c07cb848-0d0f-4f34-8b25-49a143dee3d1","Type":"ContainerStarted","Data":"c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"}
Nov 01 09:18:50.670638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.667319    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podStartSLOduration=2.667298942 podStartE2EDuration="2.667298942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:50.661670446 +0000 UTC m=+2803.856065002" watchObservedRunningTime="2026-11-01 09:18:50.667298942 +0000 UTC m=+2803.861693497"
Nov 01 09:18:50.672836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.672801    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:50.838212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.838168    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:50.988652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:50.988260    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:51.004769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:51.004733    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:51.011474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:51.011442    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:51.502493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:51.502459    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:51.503585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:51.503561    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:18:52.947723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:52.947679    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"]
Nov 01 09:18:52.951951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:52.951932    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:52.989042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:52.989009    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5jt\" (UniqueName: \"kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:52.989199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:52.989121    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:52.989256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:52.989205    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.005772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.005736    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"]
Nov 01 09:18:53.046621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.046588    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:18:53.046621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.046594    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:18:53.046822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.046645    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:18:53.049094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.049077    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:18:53.090930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.090894    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.091115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.090952    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.091159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.091118    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5jt\" (UniqueName: \"kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.091676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.091652    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.093479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.093451    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.148922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.148884    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5jt\" (UniqueName: \"kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt\") pod \"maas-api-revoke-sub-22f85aea-l7gxn\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.266333 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.266293    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:53.638492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.638463    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"]
Nov 01 09:18:53.638781 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:53.638760    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ca9908_15df_4b73_90da_2c4d9b0bb4af.slice/crio-3412a8d772c3c97e9667bad240c731934082e504cca89089682fc92b01f7163b WatchSource:0}: Error finding container 3412a8d772c3c97e9667bad240c731934082e504cca89089682fc92b01f7163b: Status 404 returned error can't find the container with id 3412a8d772c3c97e9667bad240c731934082e504cca89089682fc92b01f7163b
Nov 01 09:18:53.818919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.818883    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:53.987218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:53.987178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:54.525980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:54.525940    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn" event={"ID":"c4ca9908-15df-4b73-90da-2c4d9b0bb4af","Type":"ContainerStarted","Data":"03fdaa8ef440df24f9645062465e73aa8236050f68b2af476fcc73aa37c03367"}
Nov 01 09:18:54.526168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:54.526005    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn" event={"ID":"c4ca9908-15df-4b73-90da-2c4d9b0bb4af","Type":"ContainerStarted","Data":"3412a8d772c3c97e9667bad240c731934082e504cca89089682fc92b01f7163b"}
Nov 01 09:18:54.585604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:54.585570    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:54.636335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:54.636265    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn" podStartSLOduration=2.636250661 podStartE2EDuration="2.636250661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:54.63363885 +0000 UTC m=+2807.828033407" watchObservedRunningTime="2026-11-01 09:18:54.636250661 +0000 UTC m=+2807.830645268"
Nov 01 09:18:54.801840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:54.801758    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:54.987745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:54.987711    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:55.536949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:55.536912    4314 generic.go:356] "Generic (PLEG): container finished" podID="c4ca9908-15df-4b73-90da-2c4d9b0bb4af" containerID="03fdaa8ef440df24f9645062465e73aa8236050f68b2af476fcc73aa37c03367" exitCode=0
Nov 01 09:18:55.537139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:55.537007    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn" event={"ID":"c4ca9908-15df-4b73-90da-2c4d9b0bb4af","Type":"ContainerDied","Data":"03fdaa8ef440df24f9645062465e73aa8236050f68b2af476fcc73aa37c03367"}
Nov 01 09:18:56.199039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.199011    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:56.699557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.699507    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:56.746397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.746356    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca\") pod \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") "
Nov 01 09:18:56.746624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.746486    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt\" (UniqueName: \"kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt\") pod \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") "
Nov 01 09:18:56.746624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.746616    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca\") pod \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\" (UID: \"c4ca9908-15df-4b73-90da-2c4d9b0bb4af\") "
Nov 01 09:18:56.746990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.746966    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca" pod "c4ca9908-15df-4b73-90da-2c4d9b0bb4af" (UID: "c4ca9908-15df-4b73-90da-2c4d9b0bb4af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:18:56.747178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.747155    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:56.748660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.748634    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca" pod "c4ca9908-15df-4b73-90da-2c4d9b0bb4af" (UID: "c4ca9908-15df-4b73-90da-2c4d9b0bb4af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:18:56.748660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.748645    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt" pod "c4ca9908-15df-4b73-90da-2c4d9b0bb4af" (UID: "c4ca9908-15df-4b73-90da-2c4d9b0bb4af"). InnerVolumeSpecName "kube-api-access-8k5jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:18:56.848762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.848661    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5jt\" (UniqueName: \"kubernetes.io/projected/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-kube-api-access-8k5jt\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:56.848762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:56.848696    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ca9908-15df-4b73-90da-2c4d9b0bb4af-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:18:57.553763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:57.553719    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn" event={"ID":"c4ca9908-15df-4b73-90da-2c4d9b0bb4af","Type":"ContainerDied","Data":"3412a8d772c3c97e9667bad240c731934082e504cca89089682fc92b01f7163b"}
Nov 01 09:18:57.553763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:57.553753    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"
Nov 01 09:18:57.554182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:57.553783    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3412a8d772c3c97e9667bad240c731934082e504cca89089682fc92b01f7163b"
Nov 01 09:18:58.517145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:58.517110    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:58.688212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:58.688181    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:18:59.071675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.071635    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"]
Nov 01 09:18:59.072867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.072839    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c4ca9908-15df-4b73-90da-2c4d9b0bb4af" containerName="revoke-keys"
Nov 01 09:18:59.073062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.073027    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ca9908-15df-4b73-90da-2c4d9b0bb4af" containerName="revoke-keys"
Nov 01 09:18:59.073275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.073259    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ca9908-15df-4b73-90da-2c4d9b0bb4af" containerName="revoke-keys"
Nov 01 09:18:59.079787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.079762    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.132397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.131709    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"]
Nov 01 09:18:59.171569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.170833    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a87a1d\""
Nov 01 09:18:59.171569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.170884    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a87a1d-dockercfg-zvcr2\""
Nov 01 09:18:59.181330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.181294    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgb6l\" (UniqueName: \"kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l\") pod \"payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.181707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.181674    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.253425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.253388    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"]
Nov 01 09:18:59.257588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.257570    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.283004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.282976    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.283148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.283132    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.283198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.283176    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgb6l\" (UniqueName: \"kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l\") pod \"payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.283232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.283219    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lfv\" (UniqueName: \"kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv\") pod \"payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.283851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.283833    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.311870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.311843    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"]
Nov 01 09:18:59.340757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.340675    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgb6l\" (UniqueName: \"kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l\") pod \"payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.384738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.384708    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lfv\" (UniqueName: \"kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv\") pod \"payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.384960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.384935    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.385664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.385643    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.442415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.442380    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lfv\" (UniqueName: \"kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv\") pod \"payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.446764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.446741    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:18:59.570970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.570939    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:18:59.852714 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:59.852684    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode362b181_5253_4db8_b41a_eb53e2bde6ed.slice/crio-c1bcee3d50cac44e2b71b15591caba88c7abe574f05ee3d0a49de8267e4b1ac4 WatchSource:0}: Error finding container c1bcee3d50cac44e2b71b15591caba88c7abe574f05ee3d0a49de8267e4b1ac4: Status 404 returned error can't find the container with id c1bcee3d50cac44e2b71b15591caba88c7abe574f05ee3d0a49de8267e4b1ac4
Nov 01 09:18:59.853525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.853500    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"]
Nov 01 09:18:59.968767 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:18:59.968740    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6f5239_c5fa_4a07_b0ae_626c2be2cfae.slice/crio-2f405541b28cb3a5e7f0f1829d827d6eda3b021400313032293801f8f603b1fe WatchSource:0}: Error finding container 2f405541b28cb3a5e7f0f1829d827d6eda3b021400313032293801f8f603b1fe: Status 404 returned error can't find the container with id 2f405541b28cb3a5e7f0f1829d827d6eda3b021400313032293801f8f603b1fe
Nov 01 09:18:59.969061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:18:59.969040    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"]
Nov 01 09:19:00.581365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.581332    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" event={"ID":"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae","Type":"ContainerStarted","Data":"dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96"}
Nov 01 09:19:00.581572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.581420    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" event={"ID":"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae","Type":"ContainerStarted","Data":"2f405541b28cb3a5e7f0f1829d827d6eda3b021400313032293801f8f603b1fe"}
Nov 01 09:19:00.581625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.581567    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:19:00.582973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.582948    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" event={"ID":"e362b181-5253-4db8-b41a-eb53e2bde6ed","Type":"ContainerStarted","Data":"15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938"}
Nov 01 09:19:00.583114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.582991    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" event={"ID":"e362b181-5253-4db8-b41a-eb53e2bde6ed","Type":"ContainerStarted","Data":"c1bcee3d50cac44e2b71b15591caba88c7abe574f05ee3d0a49de8267e4b1ac4"}
Nov 01 09:19:00.583114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.583048    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:19:00.692732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.692670    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" podStartSLOduration=1.69265547 podStartE2EDuration="1.69265547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:00.689507135 +0000 UTC m=+2813.883901703" watchObservedRunningTime="2026-11-01 09:19:00.69265547 +0000 UTC m=+2813.887050024"
Nov 01 09:19:00.799717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:00.799664    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" podStartSLOduration=1.7996494950000002 podStartE2EDuration="1.799649495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:00.796887542 +0000 UTC m=+2813.991282090" watchObservedRunningTime="2026-11-01 09:19:00.799649495 +0000 UTC m=+2813.994044050"
Nov 01 09:19:01.496466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:01.496428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:04.022912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:04.022876    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:04.616829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:04.616792    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:04.802233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:04.802194    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:06.386391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:06.386348    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:08.543365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:08.543335    4314 scope.go:122] "RemoveContainer" containerID="badd8d620bc7c83a7129531cc69d08e64a1545ab1921b5e89f9ba055598e644e"
Nov 01 09:19:09.351344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:09.351301    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:09.522054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:09.522017    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:09.678490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:09.678400    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:11.371714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.371656    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:11.418487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.417618    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 09:19:11.458581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.457651    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:11.467687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.467637    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:11.495694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.495640    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:11.536570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.536161    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 09:19:11.552356 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:19:11.552268    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:19:11.561718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.561614    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 09:19:11.573671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.573613    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:11.610770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.610740    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 09:19:11.612609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.612587    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 09:19:11.714610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:11.714367    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.302594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.302518    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:12.379558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.375644    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.379558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.376519    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:19:12.422556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.418617    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.422556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.419075    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:19:12.462504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.462457    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.463082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.463056    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:19:12.468679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.468630    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.469312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.469292    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:19:12.502673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.502631    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.513030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.512909    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:19:12.533940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.533849    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.554326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.554231    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.563599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.554971    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:19:12.584932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.564418    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.584932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.564906    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:19:12.586351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.586309    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.587399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:12.587377    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:19:13.126601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.084231    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:13.422627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.378863    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:13.444452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.430859    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 09:19:13.482844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.457671    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:13.483355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.469605    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 09:19:13.492006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.469659    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:13.492006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.481676    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:19:13.518083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.517665    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:13.557694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.557622    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 09:19:13.601746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:13.601695    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:14.059800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.059765    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"]
Nov 01 09:19:14.244459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.244181    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:19:14.244725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.244707    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"]
Nov 01 09:19:14.246067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.246047    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.256801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.256767    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:19:14.259062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.259038    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:19:14.296309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.279597    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:19:14.307072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.307047    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-fd5a10\""
Nov 01 09:19:14.307895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.307873    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-fd5a10-dockercfg-qb8sr\""
Nov 01 09:19:14.308045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.307079    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:19:14.308196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.308185    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:19:14.308325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.308316    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:19:14.324513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.324489    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"]
Nov 01 09:19:14.328848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.328829    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.388556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.388206    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"]
Nov 01 09:19:14.398156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.398125    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfdd13-openshift-default-dockercfg-jf87f\""
Nov 01 09:19:14.432704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.432675    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cptx6\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.432733    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.432981    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433053    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.433261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433169    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433209    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433341    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mm5\" (UniqueName: \"kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5\") pod \"payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.433508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433484    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433713    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433889    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.433998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.433932    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.534940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.534909    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.534959    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.534998    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535020    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535074    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cptx6\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535246    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535345    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535376    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535406    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.535492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535423    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535448    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535481    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.535811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535560    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mm5\" (UniqueName: \"kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5\") pod \"payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.535811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.535690    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.536081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.536053    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.536195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.536169    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.536313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.536280    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.537669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.537648    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.538041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.538023    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.597760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.597656    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.597760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.597716    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cptx6\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.597760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.597723    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mm5\" (UniqueName: \"kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5\") pod \"payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.622291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.622263    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:14.641973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.641951    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:14.985064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:14.985031    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:15.101676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.101615    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"]
Nov 01 09:19:15.104238 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.104216    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"]
Nov 01 09:19:15.108266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.108221    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:15.108359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.108331    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:15.108396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.108382    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:15.331699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.331518    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" event={"ID":"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d","Type":"ContainerStarted","Data":"20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354"}
Nov 01 09:19:15.331699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.331614    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" event={"ID":"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d","Type":"ContainerStarted","Data":"4097d4875fbb5e058d3eaf8f2843bc370f48c1e3e86f57b69989ca9990f87c83"}
Nov 01 09:19:15.333905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.333870    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" event={"ID":"0e7f8524-5bd8-412b-94e4-3a109a8a0170","Type":"ContainerStarted","Data":"c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0"}
Nov 01 09:19:15.334057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.333920    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" event={"ID":"0e7f8524-5bd8-412b-94e4-3a109a8a0170","Type":"ContainerStarted","Data":"79b2e0eef2220e96903df9b880aca1b39fdae42776e78895bc52ceb03426be06"}
Nov 01 09:19:15.334556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.334505    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:15.450944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.450883    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podStartSLOduration=1.450865161 podStartE2EDuration="1.450865161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:15.446238928 +0000 UTC m=+2828.640633484" watchObservedRunningTime="2026-11-01 09:19:15.450865161 +0000 UTC m=+2828.645259718"
Nov 01 09:19:15.558329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.558260    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" podStartSLOduration=2.558240836 podStartE2EDuration="2.558240836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:15.554316119 +0000 UTC m=+2828.748710675" watchObservedRunningTime="2026-11-01 09:19:15.558240836 +0000 UTC m=+2828.752635404"
Nov 01 09:19:15.642444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:15.642365    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:16.078558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:16.077222    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:16.644118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:16.644083    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 01 09:19:16.644487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:16.644143    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 01 09:19:17.611046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:17.610389    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:17.643366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:17.643331    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 01 09:19:17.643590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:17.643400    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 01 09:19:18.642406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:18.642370    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 01 09:19:18.642822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:18.642423    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 01 09:19:19.643053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:19.643016    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 01 09:19:19.643443 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:19.643071    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 01 09:19:20.026117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:20.026075    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:20.642394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:20.642354    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 01 09:19:20.642626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:20.642409    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 01 09:19:21.087689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:21.087647    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:19:21.088096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:21.087699    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:21.643166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:21.643125    4314 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 01 09:19:21.643349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:21.643194    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 01 09:19:22.383558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.383467    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:22.417022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.416987    4314 generic.go:356] "Generic (PLEG): container finished" podID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerID="97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8" exitCode=0
Nov 01 09:19:22.417183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.417051    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" event={"ID":"85bc7c4b-b4b6-43ba-a76e-74ecfb931909","Type":"ContainerDied","Data":"97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"}
Nov 01 09:19:22.417511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.417490    4314 scope.go:122] "RemoveContainer" containerID="97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"
Nov 01 09:19:22.419236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.419203    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:22.419364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.419290    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:22.419364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:22.419342    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:23.025831 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.025798    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:23.026485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.026456    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:23.028302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.028277    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:19:23.399430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.399350    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"]
Nov 01 09:19:23.404482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.404461    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.425662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.425640    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" event={"ID":"85bc7c4b-b4b6-43ba-a76e-74ecfb931909","Type":"ContainerStarted","Data":"76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a"}
Nov 01 09:19:23.474556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.469051    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"]
Nov 01 09:19:23.503209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.503176    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:19:23.503455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.503433    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 01 09:19:23.503526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.503485    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 01 09:19:23.522153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522118    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522248    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522281    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522354    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522398    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522432    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522453    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnx8\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522476    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.522606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.522586    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624400    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624465    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624506    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624527    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624603    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624634    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624659    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624829    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnx8\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624880    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624896    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.624904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.624901    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.625269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.625237    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.625322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.625263    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.625365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.625319    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.627174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.627151    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.627342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.627327    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.685408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.685325    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.685571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.685521    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnx8\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:23.718425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:23.718395    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:24.182861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.182697    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"]
Nov 01 09:19:24.184943 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:19:24.184910    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7c09be_8888_4b93_86b5_99de5c94fba3.slice/crio-ca9dc5c4ea7a1c207180d9cbf1840842d163388be05df543863ebc26a56b7c47 WatchSource:0}: Error finding container ca9dc5c4ea7a1c207180d9cbf1840842d163388be05df543863ebc26a56b7c47: Status 404 returned error can't find the container with id ca9dc5c4ea7a1c207180d9cbf1840842d163388be05df543863ebc26a56b7c47
Nov 01 09:19:24.187702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.187669    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:24.187802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.187737    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:24.187802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.187771    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:19:24.434596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.434488    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" event={"ID":"7e7c09be-8888-4b93-86b5-99de5c94fba3","Type":"ContainerStarted","Data":"7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073"}
Nov 01 09:19:24.434596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.434586    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" event={"ID":"7e7c09be-8888-4b93-86b5-99de5c94fba3","Type":"ContainerStarted","Data":"ca9dc5c4ea7a1c207180d9cbf1840842d163388be05df543863ebc26a56b7c47"}
Nov 01 09:19:24.503127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.502966    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 01 09:19:24.503127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.503028    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 01 09:19:24.548337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.548268    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podStartSLOduration=1.548249963 podStartE2EDuration="1.548249963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:24.547059287 +0000 UTC m=+2837.741453842" watchObservedRunningTime="2026-11-01 09:19:24.548249963 +0000 UTC m=+2837.742644519"
Nov 01 09:19:24.719225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:24.719148    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:25.506701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:25.504940    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 01 09:19:25.506701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:25.505003    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 01 09:19:25.744733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:25.744697    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:25.745022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:25.744996    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:26.504374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:26.503922    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 01 09:19:26.504374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:26.503982    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 01 09:19:26.722013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:26.721973    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:26.722645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:26.722609    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:27.200556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.199617    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:27.267559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.267079    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:27.279558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.272615    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 09:19:27.291560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.290378    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:27.352555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.352137    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:27.390555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.385755    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:19:27.502561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.500690    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:27.512957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.507193    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 01 09:19:27.512957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.507247    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 01 09:19:27.724960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.724924    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:27.725582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:27.725518    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:28.205557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.205336    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:28.205785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.205758    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:19:28.293558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.293066    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:28.315556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.308633    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 09:19:28.315556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.310517    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:19:28.332578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.324622    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:28.332578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.325018    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:19:28.501661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.501615    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:28.503361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.502165    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:19:28.503361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.502862    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 01 09:19:28.503361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.502904    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 01 09:19:28.744107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.744061    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:28.781899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.781796    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:28.829785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.827582    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:19:28.837778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.837674    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:19:28.837778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.837759    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:19:28.854219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:28.840365    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:19:29.509707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:29.509671    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:19:29.510134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:29.510115    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:19:29.510883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:29.510859    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:19:29.718761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:29.718731    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:29.718940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:29.718783    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:30.719362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:30.719321    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:30.719846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:30.719392    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:31.719433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:31.719393    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 01 09:19:31.719965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:31.719455    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 01 09:19:32.191746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:32.191693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:33.761708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:33.761664    4314 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:19:33.762247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:33.761714    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:33.868768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:33.868735    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:33.869173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:33.869154    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:33.870255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:33.870234    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:19:37.071460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:37.071418    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:38.852123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:38.852086    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:41.325134 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:19:41.325100    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:19:41.484655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:41.484612    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:42.228220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:42.228186    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:42.403178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:42.403147    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:42.550844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:42.550811    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:46.620568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:46.620509    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"]
Nov 01 09:19:46.720692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:46.720663    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"]
Nov 01 09:19:46.966470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:46.966382    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:47.227278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:47.227200    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:47.642413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:47.642377    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:48.306773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:48.306742    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:51.620967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.620934    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh" podUID="65b76cc2-8460-4ab3-8e8f-440dfc78929f" containerName="payload-processing" containerID="cri-o://b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f" gracePeriod=25
Nov 01 09:19:51.721372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.721332    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf" podUID="24f1a00c-619d-4eec-860b-81f599a4c3e2" containerName="payload-pre-processing" containerID="cri-o://9ae066937dfae2e1a3d0652bc2d7f325d4c56ac30df3773919a8928d5d74ab0c" gracePeriod=25
Nov 01 09:19:51.895749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.895723    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:19:51.902899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.902868    4314 generic.go:356] "Generic (PLEG): container finished" podID="24f1a00c-619d-4eec-860b-81f599a4c3e2" containerID="9ae066937dfae2e1a3d0652bc2d7f325d4c56ac30df3773919a8928d5d74ab0c" exitCode=0
Nov 01 09:19:51.903107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.903077    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf" event={"ID":"24f1a00c-619d-4eec-860b-81f599a4c3e2","Type":"ContainerDied","Data":"9ae066937dfae2e1a3d0652bc2d7f325d4c56ac30df3773919a8928d5d74ab0c"}
Nov 01 09:19:51.905045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.905020    4314 generic.go:356] "Generic (PLEG): container finished" podID="65b76cc2-8460-4ab3-8e8f-440dfc78929f" containerID="b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f" exitCode=0
Nov 01 09:19:51.905368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.905107    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"
Nov 01 09:19:51.905368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.905122    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh" event={"ID":"65b76cc2-8460-4ab3-8e8f-440dfc78929f","Type":"ContainerDied","Data":"b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f"}
Nov 01 09:19:51.905368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.905173    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh" event={"ID":"65b76cc2-8460-4ab3-8e8f-440dfc78929f","Type":"ContainerDied","Data":"a08e622fd49d14e238d1436815adf8f528a89b7b7a133d25c354558d25130e9d"}
Nov 01 09:19:51.905368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.905203    4314 scope.go:122] "RemoveContainer" containerID="b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f"
Nov 01 09:19:51.922784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.922764    4314 scope.go:122] "RemoveContainer" containerID="b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f"
Nov 01 09:19:51.923089 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:19:51.923069    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f\": container with ID starting with b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f not found: ID does not exist" containerID="b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f"
Nov 01 09:19:51.923164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:51.923097    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f"} err="failed to get container status \"b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f\": rpc error: code = NotFound desc = could not find container \"b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f\": container with ID starting with b562a489740a4dff66a34e228ea43db806944a739223b267fd767abb8f57429f not found: ID does not exist"
Nov 01 09:19:52.000381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.000354    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:19:52.028318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.028273    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j\" (UniqueName: \"kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j\") pod \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") "
Nov 01 09:19:52.028570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.028520    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume\") pod \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\" (UID: \"65b76cc2-8460-4ab3-8e8f-440dfc78929f\") "
Nov 01 09:19:52.028861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.028834    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume" pod "65b76cc2-8460-4ab3-8e8f-440dfc78929f" (UID: "65b76cc2-8460-4ab3-8e8f-440dfc78929f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:19:52.029132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.029114    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b76cc2-8460-4ab3-8e8f-440dfc78929f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:19:52.030514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.030484    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j" pod "65b76cc2-8460-4ab3-8e8f-440dfc78929f" (UID: "65b76cc2-8460-4ab3-8e8f-440dfc78929f"). InnerVolumeSpecName "kube-api-access-rwt9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:19:52.130370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.130307    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume\") pod \"24f1a00c-619d-4eec-860b-81f599a4c3e2\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") "
Nov 01 09:19:52.130492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.130387    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq\" (UniqueName: \"kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq\") pod \"24f1a00c-619d-4eec-860b-81f599a4c3e2\" (UID: \"24f1a00c-619d-4eec-860b-81f599a4c3e2\") "
Nov 01 09:19:52.130717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.130694    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume" pod "24f1a00c-619d-4eec-860b-81f599a4c3e2" (UID: "24f1a00c-619d-4eec-860b-81f599a4c3e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:19:52.130877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.130860    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwt9j\" (UniqueName: \"kubernetes.io/projected/65b76cc2-8460-4ab3-8e8f-440dfc78929f-kube-api-access-rwt9j\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:19:52.130916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.130883    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f1a00c-619d-4eec-860b-81f599a4c3e2-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:19:52.132408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.132390    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq" pod "24f1a00c-619d-4eec-860b-81f599a4c3e2" (UID: "24f1a00c-619d-4eec-860b-81f599a4c3e2"). InnerVolumeSpecName "kube-api-access-s9kqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:19:52.232261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.232234    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9kqq\" (UniqueName: \"kubernetes.io/projected/24f1a00c-619d-4eec-860b-81f599a4c3e2-kube-api-access-s9kqq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:19:52.321443 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.321417    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"]
Nov 01 09:19:52.325603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.325578    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0468a5-5fb4cbd567-mcqvh"]
Nov 01 09:19:52.726278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.726250    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:52.883827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.883795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:52.914457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.914387    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"
Nov 01 09:19:52.914605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.914387    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf" event={"ID":"24f1a00c-619d-4eec-860b-81f599a4c3e2","Type":"ContainerDied","Data":"44b57791aa6c0f98c74e2832cc394234ec813924d911471df5bb1f09482edc59"}
Nov 01 09:19:52.914605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.914569    4314 scope.go:122] "RemoveContainer" containerID="9ae066937dfae2e1a3d0652bc2d7f325d4c56ac30df3773919a8928d5d74ab0c"
Nov 01 09:19:52.929191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:52.929164    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b76cc2-8460-4ab3-8e8f-440dfc78929f" path="/var/lib/kubelet/pods/65b76cc2-8460-4ab3-8e8f-440dfc78929f/volumes"
Nov 01 09:19:53.056737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:53.056698    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:53.070286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:53.070260    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"]
Nov 01 09:19:53.074257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:53.074232    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0468a5-7f849f55c9-9c2tf"]
Nov 01 09:19:54.393454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:54.393411    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:54.930112 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:54.930076    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f1a00c-619d-4eec-860b-81f599a4c3e2" path="/var/lib/kubelet/pods/24f1a00c-619d-4eec-860b-81f599a4c3e2/volumes"
Nov 01 09:19:56.510346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:56.510316    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:57.553203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.553172    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:57.611002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.610971    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"]
Nov 01 09:19:57.611686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.611665    4314 cpu_manager.go:414] "containerMap: removing container" podUID="65b76cc2-8460-4ab3-8e8f-440dfc78929f" containerName="payload-processing"
Nov 01 09:19:57.611807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.611685    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b76cc2-8460-4ab3-8e8f-440dfc78929f" containerName="payload-processing"
Nov 01 09:19:57.611807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.611742    4314 cpu_manager.go:414] "containerMap: removing container" podUID="24f1a00c-619d-4eec-860b-81f599a4c3e2" containerName="payload-pre-processing"
Nov 01 09:19:57.611807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.611752    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f1a00c-619d-4eec-860b-81f599a4c3e2" containerName="payload-pre-processing"
Nov 01 09:19:57.611974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.611881    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f1a00c-619d-4eec-860b-81f599a4c3e2" containerName="payload-pre-processing"
Nov 01 09:19:57.611974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.611899    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b76cc2-8460-4ab3-8e8f-440dfc78929f" containerName="payload-processing"
Nov 01 09:19:57.616869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.616850    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.671169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.671141    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"]
Nov 01 09:19:57.703160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.703133    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.703293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.703192    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.703416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.703399    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.711101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.711073    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:19:57.711101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.711074    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:19:57.711233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.711102    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:19:57.713395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.713379    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:19:57.804931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.804852    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.805086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.804939    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.805086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.805007    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.805687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.805659    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.807214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.807194    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.861905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.861885    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f\") pod \"maas-api-revoke-sub-14dea1ac-mh92b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.930715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.930693    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:19:57.990419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:57.990391    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:19:58.302014 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:19:58.301986    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252c5e54_29a5_4abd_b82d_54b426d3986b.slice/crio-b69b440aaedf2f84af41a184fad24a923b4e666340e779fe352233b455fcd67e WatchSource:0}: Error finding container b69b440aaedf2f84af41a184fad24a923b4e666340e779fe352233b455fcd67e: Status 404 returned error can't find the container with id b69b440aaedf2f84af41a184fad24a923b4e666340e779fe352233b455fcd67e
Nov 01 09:19:58.302767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:58.302744    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"]
Nov 01 09:19:58.967250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:58.967166    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b" event={"ID":"252c5e54-29a5-4abd-b82d-54b426d3986b","Type":"ContainerStarted","Data":"2f211f35b9abf94b0ab3b316dbe5b72be80e2f63a3ca7b33a964f17eac6cf345"}
Nov 01 09:19:58.967250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:58.967214    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b" event={"ID":"252c5e54-29a5-4abd-b82d-54b426d3986b","Type":"ContainerStarted","Data":"b69b440aaedf2f84af41a184fad24a923b4e666340e779fe352233b455fcd67e"}
Nov 01 09:19:59.080787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:59.080724    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b" podStartSLOduration=2.080706103 podStartE2EDuration="2.080706103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:59.076002715 +0000 UTC m=+2872.270397270" watchObservedRunningTime="2026-11-01 09:19:59.080706103 +0000 UTC m=+2872.275100661"
Nov 01 09:19:59.975990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:59.975957    4314 generic.go:356] "Generic (PLEG): container finished" podID="252c5e54-29a5-4abd-b82d-54b426d3986b" containerID="2f211f35b9abf94b0ab3b316dbe5b72be80e2f63a3ca7b33a964f17eac6cf345" exitCode=0
Nov 01 09:19:59.976413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:19:59.976050    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b" event={"ID":"252c5e54-29a5-4abd-b82d-54b426d3986b","Type":"ContainerDied","Data":"2f211f35b9abf94b0ab3b316dbe5b72be80e2f63a3ca7b33a964f17eac6cf345"}
Nov 01 09:20:00.412316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:00.412278    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"]
Nov 01 09:20:00.704682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:00.704588    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:00.856797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:00.856758    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:01.131252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.131219    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:20:01.248458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.248364    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f\") pod \"252c5e54-29a5-4abd-b82d-54b426d3986b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") "
Nov 01 09:20:01.248674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.248505    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca\") pod \"252c5e54-29a5-4abd-b82d-54b426d3986b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") "
Nov 01 09:20:01.248674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.248564    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca\") pod \"252c5e54-29a5-4abd-b82d-54b426d3986b\" (UID: \"252c5e54-29a5-4abd-b82d-54b426d3986b\") "
Nov 01 09:20:01.249241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.249213    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca" pod "252c5e54-29a5-4abd-b82d-54b426d3986b" (UID: "252c5e54-29a5-4abd-b82d-54b426d3986b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:20:01.250907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.250866    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f" pod "252c5e54-29a5-4abd-b82d-54b426d3986b" (UID: "252c5e54-29a5-4abd-b82d-54b426d3986b"). InnerVolumeSpecName "kube-api-access-bsd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:20:01.251026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.250968    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca" pod "252c5e54-29a5-4abd-b82d-54b426d3986b" (UID: "252c5e54-29a5-4abd-b82d-54b426d3986b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:20:01.350621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.350590    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/252c5e54-29a5-4abd-b82d-54b426d3986b-kube-api-access-bsd5f\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:01.350621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.350617    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:01.350621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.350628    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252c5e54-29a5-4abd-b82d-54b426d3986b-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:01.994709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.994657    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b" event={"ID":"252c5e54-29a5-4abd-b82d-54b426d3986b","Type":"ContainerDied","Data":"b69b440aaedf2f84af41a184fad24a923b4e666340e779fe352233b455fcd67e"}
Nov 01 09:20:01.995672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.994724    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b69b440aaedf2f84af41a184fad24a923b4e666340e779fe352233b455fcd67e"
Nov 01 09:20:01.995672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:01.994796    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"
Nov 01 09:20:03.235715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:03.235679    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:03.382794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:03.382759    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:05.413024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:05.412980    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerName="payload-pre-processing" containerID="cri-o://62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189" gracePeriod=25
Nov 01 09:20:07.295957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:07.295911    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:20:08.387203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:08.387162    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:08.580874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:08.580838    4314 scope.go:122] "RemoveContainer" containerID="9a97aea7bc8aa5ff361c28da2be51be0f21afd15c6a9828a710ae96852f5245f"
Nov 01 09:20:11.326965 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:20:11.326925    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:20:13.338111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:13.338073    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:13.498801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:13.498766    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:13.647577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:13.647470    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:16.417732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:16.417692    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:17.314687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:17.314630    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:20:18.513993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:18.513948    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:18.789690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:18.789590    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:22.708683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:22.708642    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:24.046360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:24.046315    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"]
Nov 01 09:20:24.091670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:24.091625    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8d1ab1-stplj"]
Nov 01 09:20:24.502028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:24.501991    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:24.931651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:24.931576    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e374f7-72a0-43ad-b1e9-2d20318e6735" path="/var/lib/kubelet/pods/a3e374f7-72a0-43ad-b1e9-2d20318e6735/volumes"
Nov 01 09:20:25.699769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.699734    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:20:25.788137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.788100    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume\") pod \"202cc307-c196-4bfb-8a36-f56c7c4f4967\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") "
Nov 01 09:20:25.788301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.788162    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489\" (UniqueName: \"kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489\") pod \"202cc307-c196-4bfb-8a36-f56c7c4f4967\" (UID: \"202cc307-c196-4bfb-8a36-f56c7c4f4967\") "
Nov 01 09:20:25.788572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.788547    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume" pod "202cc307-c196-4bfb-8a36-f56c7c4f4967" (UID: "202cc307-c196-4bfb-8a36-f56c7c4f4967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:20:25.788744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.788718    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202cc307-c196-4bfb-8a36-f56c7c4f4967-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:25.790588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.790562    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489" pod "202cc307-c196-4bfb-8a36-f56c7c4f4967" (UID: "202cc307-c196-4bfb-8a36-f56c7c4f4967"). InnerVolumeSpecName "kube-api-access-4h489". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:20:25.889888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:25.889836    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h489\" (UniqueName: \"kubernetes.io/projected/202cc307-c196-4bfb-8a36-f56c7c4f4967-kube-api-access-4h489\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:26.198184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.198149    4314 generic.go:356] "Generic (PLEG): container finished" podID="d73eaa51-7bee-44b5-b178-c774c7984717" containerID="32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e" exitCode=0
Nov 01 09:20:26.198410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.198229    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" event={"ID":"d73eaa51-7bee-44b5-b178-c774c7984717","Type":"ContainerDied","Data":"32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"}
Nov 01 09:20:26.198839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.198814    4314 scope.go:122] "RemoveContainer" containerID="32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"
Nov 01 09:20:26.200037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200015    4314 generic.go:356] "Generic (PLEG): container finished" podID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerID="62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189" exitCode=0
Nov 01 09:20:26.200148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200072    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"
Nov 01 09:20:26.200148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200102    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" event={"ID":"202cc307-c196-4bfb-8a36-f56c7c4f4967","Type":"ContainerDied","Data":"62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189"}
Nov 01 09:20:26.200270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200162    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486" event={"ID":"202cc307-c196-4bfb-8a36-f56c7c4f4967","Type":"ContainerDied","Data":"eec31e79cf7d5eafc09514e5a5dfafa85ef0600f923df49aa0184a15621cc1d7"}
Nov 01 09:20:26.200270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200193    4314 scope.go:122] "RemoveContainer" containerID="62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189"
Nov 01 09:20:26.200634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200599    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:26.200750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200688    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:26.200750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.200743    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:26.204253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.203984    4314 generic.go:356] "Generic (PLEG): container finished" podID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerID="c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb" exitCode=0
Nov 01 09:20:26.204253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.204037    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" event={"ID":"c07cb848-0d0f-4f34-8b25-49a143dee3d1","Type":"ContainerDied","Data":"c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"}
Nov 01 09:20:26.204574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.204556    4314 scope.go:122] "RemoveContainer" containerID="c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"
Nov 01 09:20:26.206512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.206480    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:26.206691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.206646    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:26.206691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.206685    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:26.224812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.224770    4314 scope.go:122] "RemoveContainer" containerID="62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189"
Nov 01 09:20:26.225217 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:20:26.225176    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189\": container with ID starting with 62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189 not found: ID does not exist" containerID="62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189"
Nov 01 09:20:26.225339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.225231    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189"} err="failed to get container status \"62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189\": rpc error: code = NotFound desc = could not find container \"62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189\": container with ID starting with 62ad0752ddbff56f7aa4565710a26e8d5fbbbe7452b49e0bdf4093b5154c6189 not found: ID does not exist"
Nov 01 09:20:26.423949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.423860    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"]
Nov 01 09:20:26.432830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.432788    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6752e4-b57d68575-bx486"]
Nov 01 09:20:26.930614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:26.930584    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" path="/var/lib/kubelet/pods/202cc307-c196-4bfb-8a36-f56c7c4f4967/volumes"
Nov 01 09:20:27.218049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.217948    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" event={"ID":"d73eaa51-7bee-44b5-b178-c774c7984717","Type":"ContainerStarted","Data":"f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e"}
Nov 01 09:20:27.221436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.221406    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" event={"ID":"c07cb848-0d0f-4f34-8b25-49a143dee3d1","Type":"ContainerStarted","Data":"f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7"}
Nov 01 09:20:27.812763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.812728    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:27.830757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.830717    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:20:27.831045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.830929    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 09:20:27.831045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.830991    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 09:20:27.967710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:27.967673    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:28.005042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:28.005006    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:20:28.005421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:28.005397    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 09:20:28.005615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:28.005584    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 09:20:28.830076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:28.830038    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 09:20:28.830285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:28.830100    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 09:20:29.007920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:29.007872    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 09:20:29.008440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:29.007950    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 09:20:29.470559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:29.468699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:29.830709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:29.830671    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 09:20:29.830895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:29.830743    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 09:20:30.004879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:30.004842    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 09:20:30.005152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:30.005124    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 09:20:30.010578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:30.010552    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:30.534570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:30.529767    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:30.831038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:30.830961    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 09:20:30.831272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:30.831248    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 09:20:31.007965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:31.007925    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 09:20:31.008285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:31.008255    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 09:20:31.299115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:31.299076    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:31.831006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:31.830974    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 09:20:31.831257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:31.831235    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 09:20:32.005977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:32.005934    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 09:20:32.006267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:32.006236    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 09:20:32.834932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:32.834901    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:20:32.835375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:32.835355    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:20:32.836067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:32.836045    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:20:33.008566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:33.008518    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:20:33.008822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:33.008801    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:20:33.009554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:33.009519    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 09:20:34.693198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:34.693166    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:37.290455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:37.290418    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:38.476868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:38.476834    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:38.626375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:38.626338    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:40.091261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:40.091232    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:41.322701 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:20:41.322665    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:20:43.049523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:43.049474    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"]
Nov 01 09:20:43.092910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:43.092880    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef509b1-jwxrq"]
Nov 01 09:20:43.195973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:43.195943    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:44.929008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:44.928968    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce49ef46-8906-467f-b4d1-6898b4c3cba0" path="/var/lib/kubelet/pods/ce49ef46-8906-467f-b4d1-6898b4c3cba0/volumes"
Nov 01 09:20:45.086685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.086649    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:45.699404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.699365    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"]
Nov 01 09:20:45.699996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.699980    4314 cpu_manager.go:414] "containerMap: removing container" podUID="252c5e54-29a5-4abd-b82d-54b426d3986b" containerName="revoke-keys"
Nov 01 09:20:45.700051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.699998    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252c5e54-29a5-4abd-b82d-54b426d3986b" containerName="revoke-keys"
Nov 01 09:20:45.700091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.700049    4314 cpu_manager.go:414] "containerMap: removing container" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerName="payload-pre-processing"
Nov 01 09:20:45.700091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.700056    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerName="payload-pre-processing"
Nov 01 09:20:45.700158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.700139    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="252c5e54-29a5-4abd-b82d-54b426d3986b" containerName="revoke-keys"
Nov 01 09:20:45.700158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.700153    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="202cc307-c196-4bfb-8a36-f56c7c4f4967" containerName="payload-pre-processing"
Nov 01 09:20:45.702728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.702707    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.751859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.751828    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:20:45.798079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.798043    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:20:45.798247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.798081    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:20:45.798247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.798081    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:20:45.798598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.798574    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszsg\" (UniqueName: \"kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.798670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.798655    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.798772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.798758    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.802491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.802471    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"]
Nov 01 09:20:45.899606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.899563    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.899795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.899662    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.899795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.899784    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszsg\" (UniqueName: \"kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.900194 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.900165    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.902045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.902022    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:45.957188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:45.957111    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszsg\" (UniqueName: \"kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg\") pod \"maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:46.014963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:46.014929    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:46.385136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:46.385109    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"]
Nov 01 09:20:46.385485 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:20:46.385461    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e297541_da97_4712_b1d5_82295dafbf28.slice/crio-15a1029b3025d5b55d65ec682b3b4b58dfd90aff30fa495b6d31b1d3138c09ed WatchSource:0}: Error finding container 15a1029b3025d5b55d65ec682b3b4b58dfd90aff30fa495b6d31b1d3138c09ed: Status 404 returned error can't find the container with id 15a1029b3025d5b55d65ec682b3b4b58dfd90aff30fa495b6d31b1d3138c09ed
Nov 01 09:20:46.395828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:46.395784    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s" event={"ID":"8e297541-da97-4712-b1d5-82295dafbf28","Type":"ContainerStarted","Data":"15a1029b3025d5b55d65ec682b3b4b58dfd90aff30fa495b6d31b1d3138c09ed"}
Nov 01 09:20:47.404418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:47.404383    4314 generic.go:356] "Generic (PLEG): container finished" podID="8e297541-da97-4712-b1d5-82295dafbf28" containerID="7f17bdd75c98e1cf1c84c4fb7c577abc12dacbe72b7a3ed6c74b1ae3d79930c3" exitCode=0
Nov 01 09:20:47.404418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:47.404423    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s" event={"ID":"8e297541-da97-4712-b1d5-82295dafbf28","Type":"ContainerDied","Data":"7f17bdd75c98e1cf1c84c4fb7c577abc12dacbe72b7a3ed6c74b1ae3d79930c3"}
Nov 01 09:20:48.566179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.566151    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:48.737264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.737236    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca\") pod \"8e297541-da97-4712-b1d5-82295dafbf28\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") "
Nov 01 09:20:48.737453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.737294    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca\") pod \"8e297541-da97-4712-b1d5-82295dafbf28\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") "
Nov 01 09:20:48.737453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.737437    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg\" (UniqueName: \"kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg\") pod \"8e297541-da97-4712-b1d5-82295dafbf28\" (UID: \"8e297541-da97-4712-b1d5-82295dafbf28\") "
Nov 01 09:20:48.737706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.737672    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca" pod "8e297541-da97-4712-b1d5-82295dafbf28" (UID: "8e297541-da97-4712-b1d5-82295dafbf28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:20:48.738021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.738003    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:48.739743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.739718    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca" pod "8e297541-da97-4712-b1d5-82295dafbf28" (UID: "8e297541-da97-4712-b1d5-82295dafbf28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:20:48.739904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.739890    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg" pod "8e297541-da97-4712-b1d5-82295dafbf28" (UID: "8e297541-da97-4712-b1d5-82295dafbf28"). InnerVolumeSpecName "kube-api-access-rszsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:20:48.841985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.841929    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszsg\" (UniqueName: \"kubernetes.io/projected/8e297541-da97-4712-b1d5-82295dafbf28-kube-api-access-rszsg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:48.842318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:48.842297    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e297541-da97-4712-b1d5-82295dafbf28-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:20:49.432137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:49.432104    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"
Nov 01 09:20:49.432356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:49.432138    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s" event={"ID":"8e297541-da97-4712-b1d5-82295dafbf28","Type":"ContainerDied","Data":"15a1029b3025d5b55d65ec682b3b4b58dfd90aff30fa495b6d31b1d3138c09ed"}
Nov 01 09:20:49.432356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:49.432191    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15a1029b3025d5b55d65ec682b3b4b58dfd90aff30fa495b6d31b1d3138c09ed"
Nov 01 09:20:50.118198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:50.118165    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:53.116967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:53.116893    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:53.562218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:53.562187    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:54.487205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.487169    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"]
Nov 01 09:20:54.487756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.487740    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8e297541-da97-4712-b1d5-82295dafbf28" containerName="revoke-keys"
Nov 01 09:20:54.487807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.487757    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e297541-da97-4712-b1d5-82295dafbf28" containerName="revoke-keys"
Nov 01 09:20:54.487885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.487876    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e297541-da97-4712-b1d5-82295dafbf28" containerName="revoke-keys"
Nov 01 09:20:54.491718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.491699    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520064    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520135    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520217    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520271    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520367    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520425    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr94l\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520519    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520642    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.520711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.520690    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.547589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.547522    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5b96ac-openshift-default-dockercfg-6sl4h\""
Nov 01 09:20:54.575226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.575191    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:54.591781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.591749    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"]
Nov 01 09:20:54.622043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622007    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622057    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr94l\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622172    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622217    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622257    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622421    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622472    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622519    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622586    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622606    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622649    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.622902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622833    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.623347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.622937    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.623347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.623158    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.624729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.624705    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.625048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.625028    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.718552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.716332    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr94l\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.733553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.724250    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token\") pod \"e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") " pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.805083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.804983    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:54.969614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:54.968996    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:55.230289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.230259    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"]
Nov 01 09:20:55.230570 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:20:55.230543    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94455c39_0ea2_43f4_861d_97e6bc7bfca3.slice/crio-668dee011ead20f6805dda980e2569c257706f8bcfe6d9a5f55799ef7e457043 WatchSource:0}: Error finding container 668dee011ead20f6805dda980e2569c257706f8bcfe6d9a5f55799ef7e457043: Status 404 returned error can't find the container with id 668dee011ead20f6805dda980e2569c257706f8bcfe6d9a5f55799ef7e457043
Nov 01 09:20:55.233269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.233228    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:55.233365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.233320    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:55.233406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.233362    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:20:55.413906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.413824    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:55.479887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.479795    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" event={"ID":"94455c39-0ea2-43f4-861d-97e6bc7bfca3","Type":"ContainerStarted","Data":"9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"}
Nov 01 09:20:55.479887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.479873    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" event={"ID":"94455c39-0ea2-43f4-861d-97e6bc7bfca3","Type":"ContainerStarted","Data":"668dee011ead20f6805dda980e2569c257706f8bcfe6d9a5f55799ef7e457043"}
Nov 01 09:20:55.596354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.596276    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podStartSLOduration=1.596255569 podStartE2EDuration="1.596255569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:20:55.591823335 +0000 UTC m=+2928.786217890" watchObservedRunningTime="2026-11-01 09:20:55.596255569 +0000 UTC m=+2928.790650128"
Nov 01 09:20:55.805677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:55.805587    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:56.188717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:56.188626    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:56.709200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:56.709163    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:56.811830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:56.811796    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:57.346679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:57.346652    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:57.497269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:57.497227    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:57.498248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:57.498226    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:20:57.613719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:57.613617    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:58.336327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:58.336278    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:58.508899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:58.508868    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:20:58.659485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:20:58.659396    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:00.047306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:00.047266    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"]
Nov 01 09:21:00.091785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:00.091755    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bf36bc-t7nrc"]
Nov 01 09:21:00.775737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:00.775703    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:00.929440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:00.929404    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723dc0dd-c4c6-45d7-8969-a040b1ea1da4" path="/var/lib/kubelet/pods/723dc0dd-c4c6-45d7-8969-a040b1ea1da4/volumes"
Nov 01 09:21:01.498235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.498198    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:01.665316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.665283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:01.899924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.899833    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"]
Nov 01 09:21:01.905356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.905334    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:01.957455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.957428    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"]
Nov 01 09:21:01.998644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.998595    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5b96ac\""
Nov 01 09:21:01.998918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:01.998761    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5b96ac-dockercfg-7xhlz\""
Nov 01 09:21:02.024556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.024499    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkxm\" (UniqueName: \"kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm\") pod \"payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.025023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.024996    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.126582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.126520    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkxm\" (UniqueName: \"kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm\") pod \"payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.126779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.126678    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.127406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.127381    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.183747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.183673    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkxm\" (UniqueName: \"kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm\") pod \"payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.220453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.220410    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:02.592745 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:21:02.592707    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91720349_4f67_4f40_8a1c_093a6c7b5a94.slice/crio-ff63122845ce2d4f4a96df0113b10dc4155f4a5010e3c6a29f87a7994424cec2 WatchSource:0}: Error finding container ff63122845ce2d4f4a96df0113b10dc4155f4a5010e3c6a29f87a7994424cec2: Status 404 returned error can't find the container with id ff63122845ce2d4f4a96df0113b10dc4155f4a5010e3c6a29f87a7994424cec2
Nov 01 09:21:02.594082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.594060    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"]
Nov 01 09:21:02.932108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:02.932028    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:03.211586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:03.211475    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:03.370219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:03.370187    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:03.541719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:03.541687    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" event={"ID":"91720349-4f67-4f40-8a1c-093a6c7b5a94","Type":"ContainerStarted","Data":"7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d"}
Nov 01 09:21:03.541933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:03.541737    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" event={"ID":"91720349-4f67-4f40-8a1c-093a6c7b5a94","Type":"ContainerStarted","Data":"ff63122845ce2d4f4a96df0113b10dc4155f4a5010e3c6a29f87a7994424cec2"}
Nov 01 09:21:03.541933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:03.541773    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:03.651215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:03.651153    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" podStartSLOduration=2.651135793 podStartE2EDuration="2.651135793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:03.650378199 +0000 UTC m=+2936.844772761" watchObservedRunningTime="2026-11-01 09:21:03.651135793 +0000 UTC m=+2936.845530349"
Nov 01 09:21:05.933181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:05.933148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:06.158516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.158474    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:06.360134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.360096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:06.563724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.563686    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"]
Nov 01 09:21:06.567945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.567925    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.700495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.700370    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-65aeef-dockercfg-7clwg\""
Nov 01 09:21:06.702968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.702939    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.703315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.703293    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fn2q\" (UniqueName: \"kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q\") pod \"payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.709943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.709922    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-65aeef\""
Nov 01 09:21:06.751697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.751666    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"]
Nov 01 09:21:06.805553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.805497    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.805756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.805679    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fn2q\" (UniqueName: \"kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q\") pod \"payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.806607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.806574    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.863175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.863138    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fn2q\" (UniqueName: \"kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q\") pod \"payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:06.881901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:06.881871    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:07.254045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:07.254015    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"]
Nov 01 09:21:07.254392 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:21:07.254127    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2757fbbc_4757_41f2_b7b9_da1397533569.slice/crio-1a286a759ee2dffcf3301cafe28807400b8ad95a9e8645ef095a3fff89c90fea WatchSource:0}: Error finding container 1a286a759ee2dffcf3301cafe28807400b8ad95a9e8645ef095a3fff89c90fea: Status 404 returned error can't find the container with id 1a286a759ee2dffcf3301cafe28807400b8ad95a9e8645ef095a3fff89c90fea
Nov 01 09:21:07.572049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:07.571965    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" event={"ID":"2757fbbc-4757-41f2-b7b9-da1397533569","Type":"ContainerStarted","Data":"569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b"}
Nov 01 09:21:07.572049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:07.572017    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" event={"ID":"2757fbbc-4757-41f2-b7b9-da1397533569","Type":"ContainerStarted","Data":"1a286a759ee2dffcf3301cafe28807400b8ad95a9e8645ef095a3fff89c90fea"}
Nov 01 09:21:07.572265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:07.572182    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:07.681505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:07.681452    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" podStartSLOduration=1.6814381969999999 podStartE2EDuration="1.681438197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:07.680435509 +0000 UTC m=+2940.874830063" watchObservedRunningTime="2026-11-01 09:21:07.681438197 +0000 UTC m=+2940.875832751"
Nov 01 09:21:11.210163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:11.210127    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:11.325846 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:21:11.325811    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:21:13.047502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:13.047468    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"]
Nov 01 09:21:13.101683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:13.096036    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b6d862d-q6xq4"]
Nov 01 09:21:14.551574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:14.551526    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:21:14.929674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:14.929598    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8c3674-de68-4eea-a396-b8723f06a920" path="/var/lib/kubelet/pods/1e8c3674-de68-4eea-a396-b8723f06a920/volumes"
Nov 01 09:21:16.080900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:16.080868    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:16.757551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:16.757510    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:16.904821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:16.904792    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:17.097066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.096980    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"]
Nov 01 09:21:17.142846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.142807    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06908f1d-ws24j"]
Nov 01 09:21:17.268647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.268606    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:17.758018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.757983    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"]
Nov 01 09:21:17.763680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.763655    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.813599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.813568    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-46bf99-openshift-default-dockercfg-2rzjn\""
Nov 01 09:21:17.904555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.895248    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.904963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.904936    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.919442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.919404    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.919750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.919727    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.919880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.919866    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.920000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.919986    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.920200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.920187    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.920466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.920448    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.920720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.920699    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7t5h\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:17.921745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.921677    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:17.928401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:17.928325    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"]
Nov 01 09:21:18.022474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022379    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.022474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022440    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.022474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022471    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.022790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022505    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.022790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022525    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.022790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022565    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.022790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022656    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022790    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022820    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7t5h\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.022995    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.023054    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.023113    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.023174    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.023453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.023431    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.025339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.025308    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.025654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.025634    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.079638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.079605    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.079972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.079949    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7t5h\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.215695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.215658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:18.377446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.377365    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:18.582207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.582178    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 09:21:18.846551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.846512    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"]
Nov 01 09:21:18.847805 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:21:18.847768    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823fb4f0_2d4a_4e2d_ac37_3c03749efbf1.slice/crio-a5675fb01e5754539a75b38a64e89eff0b2eb4f58bd8e732cd6467a236f6666d WatchSource:0}: Error finding container a5675fb01e5754539a75b38a64e89eff0b2eb4f58bd8e732cd6467a236f6666d: Status 404 returned error can't find the container with id a5675fb01e5754539a75b38a64e89eff0b2eb4f58bd8e732cd6467a236f6666d
Nov 01 09:21:18.850906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.850867    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:21:18.851022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.850951    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:21:18.851022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.851000    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:21:18.929571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:18.929549    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e3e35b-5bc8-43d7-b492-20fe8a2f9115" path="/var/lib/kubelet/pods/09e3e35b-5bc8-43d7-b492-20fe8a2f9115/volumes"
Nov 01 09:21:19.666497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:19.666456    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" event={"ID":"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1","Type":"ContainerStarted","Data":"6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"}
Nov 01 09:21:19.666928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:19.666516    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" event={"ID":"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1","Type":"ContainerStarted","Data":"a5675fb01e5754539a75b38a64e89eff0b2eb4f58bd8e732cd6467a236f6666d"}
Nov 01 09:21:19.873104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:19.873052    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podStartSLOduration=2.873037505 podStartE2EDuration="2.873037505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:19.825027351 +0000 UTC m=+2953.019421907" watchObservedRunningTime="2026-11-01 09:21:19.873037505 +0000 UTC m=+2953.067432060"
Nov 01 09:21:20.377938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:20.377904    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:20.383303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:20.383276    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:20.674340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:20.674240    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:20.675466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:20.675446    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:21:21.400736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:21.400705    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:22.623929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:22.623897    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:22.796222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:22.796190    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:22.961163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:22.961079    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:24.053860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:24.053828    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"]
Nov 01 09:21:24.099858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:24.099826    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e444f639-25wb6"]
Nov 01 09:21:24.929688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:24.929654    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b5b977-2e11-485c-8d15-a83f509df500" path="/var/lib/kubelet/pods/72b5b977-2e11-485c-8d15-a83f509df500/volumes"
Nov 01 09:21:25.746237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:25.746203    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:25.929506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:25.929428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:26.805828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:26.805792    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:27.046516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:27.046482    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"]
Nov 01 09:21:27.097809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:27.097713    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5333af69-ltv8z"]
Nov 01 09:21:28.929268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:28.929238    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25209b9a-10d8-4903-9202-1ec041b5dab3" path="/var/lib/kubelet/pods/25209b9a-10d8-4903-9202-1ec041b5dab3/volumes"
Nov 01 09:21:31.239727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:31.239695    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:32.814524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:32.814485    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:33.259547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:33.259497    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:34.228698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:34.228664    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:34.381652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:34.381612    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:35.750009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:35.749975    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:36.318164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:36.318132    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:36.319647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:36.319624    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"]
Nov 01 09:21:36.319859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:36.319838    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy" containerID="cri-o://76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a" gracePeriod=30
Nov 01 09:21:38.627900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:38.627869    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:40.046434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:40.046400    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"]
Nov 01 09:21:40.092391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:40.092359    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe6e194-9z58d"]
Nov 01 09:21:40.929690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:40.929657    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdee6922-4d40-447c-a174-ca9281b0304d" path="/var/lib/kubelet/pods/fdee6922-4d40-447c-a174-ca9281b0304d/volumes"
Nov 01 09:21:41.326014 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:21:41.325976    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:21:41.591906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.591836    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:21:41.736026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.735986    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736030    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736063    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736118    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736220    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736281    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736286    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:41.736425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736319    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736489    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:41.736627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736507    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736516    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:41.736627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736582    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo\") pod \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\" (UID: \"85bc7c4b-b4b6-43ba-a76e-74ecfb931909\") "
Nov 01 09:21:41.736839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736631    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:21:41.736839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.736752    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:41.737133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.737114    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.737133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.737133    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.737320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.737142    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.737320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.737151    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.737320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.737162    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.738664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.738640    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:21:41.738814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.738767    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "kube-api-access-vdwkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:41.738982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.738950    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:41.738982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.738960    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token" pod "85bc7c4b-b4b6-43ba-a76e-74ecfb931909" (UID: "85bc7c4b-b4b6-43ba-a76e-74ecfb931909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:41.838900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.838864    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.838900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.838902    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.839181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.838918    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.839181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.838933    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdwkg\" (UniqueName: \"kubernetes.io/projected/85bc7c4b-b4b6-43ba-a76e-74ecfb931909-kube-api-access-vdwkg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:41.844806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.844734    4314 generic.go:356] "Generic (PLEG): container finished" podID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerID="76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a" exitCode=0
Nov 01 09:21:41.844806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.844790    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"
Nov 01 09:21:41.845005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.844807    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" event={"ID":"85bc7c4b-b4b6-43ba-a76e-74ecfb931909","Type":"ContainerDied","Data":"76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a"}
Nov 01 09:21:41.845005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.844868    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m" event={"ID":"85bc7c4b-b4b6-43ba-a76e-74ecfb931909","Type":"ContainerDied","Data":"1b0ad9a0bab3f7f404bcba3f5ac72d4538f5844644d670e8a2c3c347bb9d0164"}
Nov 01 09:21:41.845005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.844892    4314 scope.go:122] "RemoveContainer" containerID="76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a"
Nov 01 09:21:41.868948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.868923    4314 scope.go:122] "RemoveContainer" containerID="97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"
Nov 01 09:21:41.879187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.879165    4314 scope.go:122] "RemoveContainer" containerID="76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a"
Nov 01 09:21:41.879479 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:21:41.879458    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a\": container with ID starting with 76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a not found: ID does not exist" containerID="76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a"
Nov 01 09:21:41.879560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.879489    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a"} err="failed to get container status \"76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a\": rpc error: code = NotFound desc = could not find container \"76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a\": container with ID starting with 76010135745f4e289b94de667ebb2d3d877e197fa7d4b8a9112d15e40f70956a not found: ID does not exist"
Nov 01 09:21:41.879560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.879511    4314 scope.go:122] "RemoveContainer" containerID="97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"
Nov 01 09:21:41.879774 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:21:41.879756    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8\": container with ID starting with 97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8 not found: ID does not exist" containerID="97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"
Nov 01 09:21:41.879821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:41.879781    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8"} err="failed to get container status \"97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8\": rpc error: code = NotFound desc = could not find container \"97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8\": container with ID starting with 97e90f6f6fff96ad6d5eda5b7280219686c166fc932e20dee1f889e683f801e8 not found: ID does not exist"
Nov 01 09:21:42.001883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:42.001849    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"]
Nov 01 09:21:42.009633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:42.009599    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3b7524-openshift-default-7dfb8c84f7-x8k4m"]
Nov 01 09:21:42.928890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:42.928858    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" path="/var/lib/kubelet/pods/85bc7c4b-b4b6-43ba-a76e-74ecfb931909/volumes"
Nov 01 09:21:43.365401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:43.365370    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:44.047002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:44.046971    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"]
Nov 01 09:21:44.093974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:44.093936    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c672939b-zk89l"]
Nov 01 09:21:44.929234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:44.929200    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb71789-d205-46a9-b636-1df2dba34a9e" path="/var/lib/kubelet/pods/cdb71789-d205-46a9-b636-1df2dba34a9e/volumes"
Nov 01 09:21:45.077705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.077634    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:45.227705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.227608    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:45.377210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.377170    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:45.567739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.567702    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"]
Nov 01 09:21:45.614758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.614725    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6752e4-bhg8s"]
Nov 01 09:21:45.641922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.641889    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:45.795214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.795175    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:45.805659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.805628    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"]
Nov 01 09:21:45.805933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.805910    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy" containerID="cri-o://90abec001d136c82a16c65ff6a9cf709d5123fdbafdcf6f76802d60a9497249e" gracePeriod=30
Nov 01 09:21:45.949746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:45.949623    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:46.641803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:46.641764    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:46.929153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:46.929076    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e297541-da97-4712-b1d5-82295dafbf28" path="/var/lib/kubelet/pods/8e297541-da97-4712-b1d5-82295dafbf28/volumes"
Nov 01 09:21:48.524643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.524603    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"]
Nov 01 09:21:48.525435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.525410    4314 cpu_manager.go:414] "containerMap: removing container" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy"
Nov 01 09:21:48.525579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.525436    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy"
Nov 01 09:21:48.525579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.525457    4314 cpu_manager.go:414] "containerMap: removing container" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy"
Nov 01 09:21:48.525579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.525464    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy"
Nov 01 09:21:48.525757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.525625    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy"
Nov 01 09:21:48.530761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.530739    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.580653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.579406    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:21:48.582290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.582264    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"]
Nov 01 09:21:48.595816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.595267    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:21:48.595816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.595342    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:21:48.623200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.623168    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:21:48.635476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.635448    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7nw\" (UniqueName: \"kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.635642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.635509    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.635749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.635728    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.736711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.736674    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.736879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.736820    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7nw\" (UniqueName: \"kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.736879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.736869    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.737341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.737315    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.739151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.739131    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.794430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.794342    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7nw\" (UniqueName: \"kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw\") pod \"maas-api-revoke-sub-047a31d7-vm8w9\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:48.844408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:48.844377    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:49.216208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:49.216169    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"]
Nov 01 09:21:49.216363 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:21:49.216176    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18def52f_4ef8_4e49_8d8b_da5ea74f3bd3.slice/crio-2324d4621d6f952324890f42ce26ac6354195aad81923483e1980ecacd41e62d WatchSource:0}: Error finding container 2324d4621d6f952324890f42ce26ac6354195aad81923483e1980ecacd41e62d: Status 404 returned error can't find the container with id 2324d4621d6f952324890f42ce26ac6354195aad81923483e1980ecacd41e62d
Nov 01 09:21:49.916050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:49.916013    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9" event={"ID":"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3","Type":"ContainerStarted","Data":"41e55692e65aeaece951e9e8e926daef768129b05423fbb0a9875b8d1dbcc9df"}
Nov 01 09:21:49.916483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:49.916079    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9" event={"ID":"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3","Type":"ContainerStarted","Data":"2324d4621d6f952324890f42ce26ac6354195aad81923483e1980ecacd41e62d"}
Nov 01 09:21:50.026754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.026687    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9" podStartSLOduration=2.026674106 podStartE2EDuration="2.026674106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:50.023660446 +0000 UTC m=+2983.218055002" watchObservedRunningTime="2026-11-01 09:21:50.026674106 +0000 UTC m=+2983.221068660"
Nov 01 09:21:50.395159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.395128    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:50.927300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.927258    4314 generic.go:356] "Generic (PLEG): container finished" podID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerID="90abec001d136c82a16c65ff6a9cf709d5123fdbafdcf6f76802d60a9497249e" exitCode=0
Nov 01 09:21:50.929528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.929503    4314 generic.go:356] "Generic (PLEG): container finished" podID="18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" containerID="41e55692e65aeaece951e9e8e926daef768129b05423fbb0a9875b8d1dbcc9df" exitCode=0
Nov 01 09:21:50.931883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.931858    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" event={"ID":"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f","Type":"ContainerDied","Data":"90abec001d136c82a16c65ff6a9cf709d5123fdbafdcf6f76802d60a9497249e"}
Nov 01 09:21:50.931977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.931915    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9" event={"ID":"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3","Type":"ContainerDied","Data":"41e55692e65aeaece951e9e8e926daef768129b05423fbb0a9875b8d1dbcc9df"}
Nov 01 09:21:50.931977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:50.931945    4314 scope.go:122] "RemoveContainer" containerID="3bd022a8e6d6be4c87b232f62957af32e31ce44fc4e57509feae71ae0c7e49fe"
Nov 01 09:21:51.095665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.095636    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:21:51.167377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167342    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.167574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167413    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.167574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167460    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.167574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167509    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.167776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167588    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.167776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167672    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.167776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167681    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:51.167776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167692    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:51.167776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167770    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.168096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167828    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.168096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167859    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert\") pod \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\" (UID: \"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f\") "
Nov 01 09:21:51.168096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167878    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:51.168096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.167892    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:51.168260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.168227    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:21:51.168699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.168671    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.168781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.168705    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.168781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.168722    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.168781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.168738    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.168781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.168752    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.169861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.169833    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:51.169861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.169843    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:21:51.170273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.170245    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:21:51.170273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.170252    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs" pod "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" (UID: "7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f"). InnerVolumeSpecName "kube-api-access-gntcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:51.269815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.269777    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntcs\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-kube-api-access-gntcs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.269815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.269812    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.270033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.269825    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.270033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.269839    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:51.938838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.938804    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"
Nov 01 09:21:51.939317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.938838    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw" event={"ID":"7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f","Type":"ContainerDied","Data":"57ef1da670dc67d53dbe242cb3d78764a6275cec5d950af4bded01272867e804"}
Nov 01 09:21:51.939317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:51.938906    4314 scope.go:122] "RemoveContainer" containerID="90abec001d136c82a16c65ff6a9cf709d5123fdbafdcf6f76802d60a9497249e"
Nov 01 09:21:52.096619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.096593    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:52.140832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.140797    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"]
Nov 01 09:21:52.146796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.146768    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-hd9kw"]
Nov 01 09:21:52.181599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.181562    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca\") pod \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") "
Nov 01 09:21:52.182900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.182171    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw\" (UniqueName: \"kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw\") pod \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") "
Nov 01 09:21:52.182900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.182329    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca\") pod \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\" (UID: \"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3\") "
Nov 01 09:21:52.183108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.182933    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca" pod "18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" (UID: "18def52f-4ef8-4e49-8d8b-da5ea74f3bd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:21:52.183559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.183515    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:52.184206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.184178    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca" pod "18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" (UID: "18def52f-4ef8-4e49-8d8b-da5ea74f3bd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:21:52.184740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.184720    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw" pod "18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" (UID: "18def52f-4ef8-4e49-8d8b-da5ea74f3bd3"). InnerVolumeSpecName "kube-api-access-nw7nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:52.284690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.284655    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:52.284690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.284693    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7nw\" (UniqueName: \"kubernetes.io/projected/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3-kube-api-access-nw7nw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:52.574664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.574622    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:52.930096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.930018    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" path="/var/lib/kubelet/pods/7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f/volumes"
Nov 01 09:21:52.946943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.946912    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"
Nov 01 09:21:52.946943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.946936    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9" event={"ID":"18def52f-4ef8-4e49-8d8b-da5ea74f3bd3","Type":"ContainerDied","Data":"2324d4621d6f952324890f42ce26ac6354195aad81923483e1980ecacd41e62d"}
Nov 01 09:21:52.947352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:52.946983    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2324d4621d6f952324890f42ce26ac6354195aad81923483e1980ecacd41e62d"
Nov 01 09:21:53.421294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:53.421256    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"]
Nov 01 09:21:53.520640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:53.520604    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"]
Nov 01 09:21:53.850423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:53.850387    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:54.037940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:54.037905    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:55.404910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:55.404872    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:55.645782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:55.645749    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:21:58.422590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.422488    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz" podUID="61b24a2a-9405-442d-a4ef-afa830ce8aab" containerName="payload-processing" containerID="cri-o://18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c" gracePeriod=25
Nov 01 09:21:58.521852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.521809    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf" podUID="44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" containerName="payload-pre-processing" containerID="cri-o://3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b" gracePeriod=25
Nov 01 09:21:58.725974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.725946    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:21:58.815033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.815008    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:21:58.910986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.910953    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s\" (UniqueName: \"kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s\") pod \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") "
Nov 01 09:21:58.911185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.911000    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj\" (UniqueName: \"kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj\") pod \"61b24a2a-9405-442d-a4ef-afa830ce8aab\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") "
Nov 01 09:21:58.911185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.911166    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume\") pod \"61b24a2a-9405-442d-a4ef-afa830ce8aab\" (UID: \"61b24a2a-9405-442d-a4ef-afa830ce8aab\") "
Nov 01 09:21:58.911322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.911256    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume\") pod \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\" (UID: \"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41\") "
Nov 01 09:21:58.911772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.911744    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume" pod "61b24a2a-9405-442d-a4ef-afa830ce8aab" (UID: "61b24a2a-9405-442d-a4ef-afa830ce8aab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:21:58.911960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.911840    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume" pod "44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" (UID: "44fa2b66-90c1-4f83-bcb2-82e9e9e63b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:21:58.913379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.913348    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s" pod "44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" (UID: "44fa2b66-90c1-4f83-bcb2-82e9e9e63b41"). InnerVolumeSpecName "kube-api-access-7r75s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:58.913379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.913372    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj" pod "61b24a2a-9405-442d-a4ef-afa830ce8aab" (UID: "61b24a2a-9405-442d-a4ef-afa830ce8aab"). InnerVolumeSpecName "kube-api-access-gp2qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:21:58.997328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.997293    4314 generic.go:356] "Generic (PLEG): container finished" podID="61b24a2a-9405-442d-a4ef-afa830ce8aab" containerID="18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c" exitCode=0
Nov 01 09:21:58.997508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.997366    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"
Nov 01 09:21:58.997508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.997386    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz" event={"ID":"61b24a2a-9405-442d-a4ef-afa830ce8aab","Type":"ContainerDied","Data":"18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c"}
Nov 01 09:21:58.997508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.997455    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz" event={"ID":"61b24a2a-9405-442d-a4ef-afa830ce8aab","Type":"ContainerDied","Data":"b53c1cc4fcd16fb0a88077ad131848510a23a66c363d4d58890b6d912b9a032c"}
Nov 01 09:21:58.997508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.997485    4314 scope.go:122] "RemoveContainer" containerID="18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c"
Nov 01 09:21:58.998887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.998856    4314 generic.go:356] "Generic (PLEG): container finished" podID="44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" containerID="3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b" exitCode=0
Nov 01 09:21:58.999062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.998909    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf" event={"ID":"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41","Type":"ContainerDied","Data":"3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b"}
Nov 01 09:21:58.999062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.998964    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf" event={"ID":"44fa2b66-90c1-4f83-bcb2-82e9e9e63b41","Type":"ContainerDied","Data":"11841662b13967a5bef9a4a8a8bd916a09ce4322f49e83d9d3ff199b660de165"}
Nov 01 09:21:58.999062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:58.998924    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"
Nov 01 09:21:59.008698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.008678    4314 scope.go:122] "RemoveContainer" containerID="18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c"
Nov 01 09:21:59.009849 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:21:59.009693    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c\": container with ID starting with 18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c not found: ID does not exist" containerID="18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c"
Nov 01 09:21:59.009849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.009733    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c"} err="failed to get container status \"18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c\": rpc error: code = NotFound desc = could not find container \"18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c\": container with ID starting with 18e96de0bcbc83167f7b3eb7b22c1549d46591147332496a4cbb48cc7f58c89c not found: ID does not exist"
Nov 01 09:21:59.009849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.009756    4314 scope.go:122] "RemoveContainer" containerID="3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b"
Nov 01 09:21:59.013330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.013305    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b24a2a-9405-442d-a4ef-afa830ce8aab-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:59.013330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.013330    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:59.013507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.013341    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r75s\" (UniqueName: \"kubernetes.io/projected/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41-kube-api-access-7r75s\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:59.013507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.013350    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp2qj\" (UniqueName: \"kubernetes.io/projected/61b24a2a-9405-442d-a4ef-afa830ce8aab-kube-api-access-gp2qj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:21:59.020368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.020343    4314 scope.go:122] "RemoveContainer" containerID="3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b"
Nov 01 09:21:59.020671 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:21:59.020652    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b\": container with ID starting with 3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b not found: ID does not exist" containerID="3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b"
Nov 01 09:21:59.020780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.020678    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b"} err="failed to get container status \"3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b\": rpc error: code = NotFound desc = could not find container \"3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b\": container with ID starting with 3881e332b75defbfe99fd25e1ce3005c20da108c3fc782ba3ad91d86db34d61b not found: ID does not exist"
Nov 01 09:21:59.116035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.115999    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"]
Nov 01 09:21:59.122849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.122824    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514eb3-5ff74c68f-55hxz"]
Nov 01 09:21:59.226122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.226050    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"]
Nov 01 09:21:59.231376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.231357    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514eb3-84476df85f-lxjhf"]
Nov 01 09:21:59.525693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:21:59.525657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:00.727651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:00.727620    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:00.887418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:00.887384    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:00.929316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:00.929286    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" path="/var/lib/kubelet/pods/44fa2b66-90c1-4f83-bcb2-82e9e9e63b41/volumes"
Nov 01 09:22:00.929747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:00.929734    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b24a2a-9405-442d-a4ef-afa830ce8aab" path="/var/lib/kubelet/pods/61b24a2a-9405-442d-a4ef-afa830ce8aab/volumes"
Nov 01 09:22:01.085565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:01.085514    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"]
Nov 01 09:22:01.093673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:01.093639    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06218628-fxvzp"]
Nov 01 09:22:01.111323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:01.111295    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:02.937066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:02.937026    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7c4d40-b881-45d2-84c0-e16becfca818" path="/var/lib/kubelet/pods/7a7c4d40-b881-45d2-84c0-e16becfca818/volumes"
Nov 01 09:22:05.068772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:05.068740    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:06.930576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:06.930547    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:08.050523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:08.050489    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"]
Nov 01 09:22:08.097006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:08.096961    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ff1694-7hxwn"]
Nov 01 09:22:08.651157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:08.651124    4314 scope.go:122] "RemoveContainer" containerID="d6fe76df9774520e59a62192e22ba2d38f5e417f5938dabc8bf6caa93102f9d3"
Nov 01 09:22:08.661097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:08.661059    4314 scope.go:122] "RemoveContainer" containerID="29f197fe442f3ffc036d7ddab162417fa2d91dc02cfa585802b84eef9363a4e8"
Nov 01 09:22:08.670345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:08.670325    4314 scope.go:122] "RemoveContainer" containerID="aae730907fa4c7160a53db17ebbdf6fe9f85d8019b487dcd3dc891432be46fed"
Nov 01 09:22:08.929769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:08.929686    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df58d2c-fd43-499f-ab6c-e6eb4fb21967" path="/var/lib/kubelet/pods/2df58d2c-fd43-499f-ab6c-e6eb4fb21967/volumes"
Nov 01 09:22:10.660846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:10.660812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:11.325035 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:22:11.324996    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:22:11.701184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:11.701098    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:15.694023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:15.693989    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:15.846738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:15.846702    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:16.627743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:16.627709    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:17.094513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:17.094484    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:17.742425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:17.742391    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"]
Nov 01 09:22:17.742626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:17.742597    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" containerID="cri-o://d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae" gracePeriod=30
Nov 01 09:22:18.537566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:18.537498    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:18.951055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:18.950958    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:20.110460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:20.110427    4314 patch_prober.go:29] interesting pod/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:22:20.110894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:20.110489    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:22:21.162216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162185    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"]
Nov 01 09:22:21.162797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162780    4314 cpu_manager.go:414] "containerMap: removing container" podUID="44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" containerName="payload-pre-processing"
Nov 01 09:22:21.162853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162799    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" containerName="payload-pre-processing"
Nov 01 09:22:21.162853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162820    4314 cpu_manager.go:414] "containerMap: removing container" podUID="61b24a2a-9405-442d-a4ef-afa830ce8aab" containerName="payload-processing"
Nov 01 09:22:21.162853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162825    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b24a2a-9405-442d-a4ef-afa830ce8aab" containerName="payload-processing"
Nov 01 09:22:21.162853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162839    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy"
Nov 01 09:22:21.162853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162844    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy"
Nov 01 09:22:21.162853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162853    4314 cpu_manager.go:414] "containerMap: removing container" podUID="18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" containerName="revoke-keys"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162858    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" containerName="revoke-keys"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162871    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162875    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162966    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b24a2a-9405-442d-a4ef-afa830ce8aab" containerName="payload-processing"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162978    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162984    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" containerName="revoke-keys"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.162991    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bc7c4b-b4b6-43ba-a76e-74ecfb931909" containerName="istio-proxy"
Nov 01 09:22:21.163047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.163000    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fa2b66-90c1-4f83-bcb2-82e9e9e63b41" containerName="payload-pre-processing"
Nov 01 09:22:21.168686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.168669    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.228333 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.228302    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-58de5a-openshift-default-dockercfg-tfsb7\""
Nov 01 09:22:21.250182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.250147    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:21.269869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.269837    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"]
Nov 01 09:22:21.365047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.352440    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.377553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.369679    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.377949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.377921    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.378151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.378135    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.386598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.378317    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.386907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.386883    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.387080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.387063    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.387249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.387234    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.387408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.387391    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68p8t\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504752    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504819    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504849    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504877    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504955    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.504995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.504983    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505014    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505045    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68p8t\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505289    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505333    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505391    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505441    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.505803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.505777    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.507562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.507525    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.507740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.507720    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.562606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.562574    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.562738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.562638    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68p8t\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t\") pod \"e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") " pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.599198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.599164    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:21.781885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.781803    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:21.936816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:21.936784    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:22.152300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.152195    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:22.260357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.256931    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"]
Nov 01 09:22:22.260357 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:22:22.258109    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1910e0cc_1bef_4ef4_96cb_0b7ddc7167e5.slice/crio-23ee5dc16db5e368cfecd50de2370322cedf0979b86d1ede27c59e4d27e2774f WatchSource:0}: Error finding container 23ee5dc16db5e368cfecd50de2370322cedf0979b86d1ede27c59e4d27e2774f: Status 404 returned error can't find the container with id 23ee5dc16db5e368cfecd50de2370322cedf0979b86d1ede27c59e4d27e2774f
Nov 01 09:22:22.262222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.262180    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:22.262330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.262279    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:22.262330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.262323    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:22.295401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.295376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:22.681728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.681695    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"]
Nov 01 09:22:22.682440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.682423    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6f7bd4-57ac-4fa6-9f1b-b5b7f010d07f" containerName="istio-proxy"
Nov 01 09:22:22.686111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.686089    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.736188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.736148    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-e40e82-openshift-default-dockercfg-qr4vb\""
Nov 01 09:22:22.744583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.744554    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"]
Nov 01 09:22:22.762771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.762726    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.762917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.762797    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.762917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.762888    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.763025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.762960    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.763025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.763019    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.763137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.763089    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.763193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.763156    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.763193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.763179    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.763299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.763236    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcxd\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.813107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.813079    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:22.864379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864344    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864444    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864513    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864613    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864687    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864714    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864774    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcxd\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864822    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864871    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864894    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.864989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.864940    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.865204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.865009    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.865266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.865213    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.865328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.865303    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.867043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.867010    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.867214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.867196    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.922980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.922943    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.925269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.925236    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcxd\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd\") pod \"e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") " pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:22.970276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:22.970083    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:23.010387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.010352    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:23.089245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.088156    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:22:23.112426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112397    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.112592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112484    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.112592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112577    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.112688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112652    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.112726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112688    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.112758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112738    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.113095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112813    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.113095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112889    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.113095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.112920    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert\") pod \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\" (UID: \"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0\") "
Nov 01 09:22:23.113374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113274    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:22:23.113374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113292    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:22:23.113479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113461    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:22:23.113659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113644    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.113704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113661    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.113704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113671    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.113704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113671    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:22:23.113792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.113718    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:22:23.115400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.115375    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:22:23.115995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.115978    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "kube-api-access-c7nbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:22:23.116304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.116265    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:22:23.119023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.119000    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy" pod "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" (UID: "23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:22:23.177163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.177126    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" event={"ID":"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5","Type":"ContainerStarted","Data":"43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"}
Nov 01 09:22:23.177364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.177189    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" event={"ID":"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5","Type":"ContainerStarted","Data":"23ee5dc16db5e368cfecd50de2370322cedf0979b86d1ede27c59e4d27e2774f"}
Nov 01 09:22:23.179139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.179109    4314 generic.go:356] "Generic (PLEG): container finished" podID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerID="d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae" exitCode=0
Nov 01 09:22:23.179263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.179167    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"
Nov 01 09:22:23.179263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.179195    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" event={"ID":"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0","Type":"ContainerDied","Data":"d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae"}
Nov 01 09:22:23.179263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.179256    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4" event={"ID":"23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0","Type":"ContainerDied","Data":"d478b634e505944b367afc863c04242ea0dd4f9fa10580f795d2ea0a4946d1d5"}
Nov 01 09:22:23.179389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.179289    4314 scope.go:122] "RemoveContainer" containerID="d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae"
Nov 01 09:22:23.193241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.193214    4314 scope.go:122] "RemoveContainer" containerID="55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"
Nov 01 09:22:23.204959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.204209    4314 scope.go:122] "RemoveContainer" containerID="d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae"
Nov 01 09:22:23.206362 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:22:23.206312    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae\": container with ID starting with d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae not found: ID does not exist" containerID="d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae"
Nov 01 09:22:23.206477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.206353    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae"} err="failed to get container status \"d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae\": rpc error: code = NotFound desc = could not find container \"d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae\": container with ID starting with d7ae7a13f9b2a829e974a60bbe2708340c0c1fd1a92db0faf5645176294633ae not found: ID does not exist"
Nov 01 09:22:23.206477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.206377    4314 scope.go:122] "RemoveContainer" containerID="55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"
Nov 01 09:22:23.206767 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:22:23.206740    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa\": container with ID starting with 55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa not found: ID does not exist" containerID="55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"
Nov 01 09:22:23.206849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.206775    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa"} err="failed to get container status \"55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa\": rpc error: code = NotFound desc = could not find container \"55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa\": container with ID starting with 55937b62e46844fbfaa2125a0d39f3ac685e31f3130d904f7d2e3c043c0b1eaa not found: ID does not exist"
Nov 01 09:22:23.215086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.215059    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.215198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.215093    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.215198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.215112    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.215198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.215128    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.215198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.215144    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.215198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.215165    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7nbg\" (UniqueName: \"kubernetes.io/projected/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0-kube-api-access-c7nbg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:22:23.313648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.313615    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"]
Nov 01 09:22:23.315794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.315758    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-514eb3-openshift-default-59bdcb95bf-hwcd4"]
Nov 01 09:22:23.427652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.427595    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podStartSLOduration=2.427577344 podStartE2EDuration="2.427577344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:23.42378006 +0000 UTC m=+3016.618174620" watchObservedRunningTime="2026-11-01 09:22:23.427577344 +0000 UTC m=+3016.621971903"
Nov 01 09:22:23.441725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.441693    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"]
Nov 01 09:22:23.442023 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:22:23.442001    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbea5e9_84f4_4f79_a929_8e86d07d4d6f.slice/crio-8af4eb0084ecd75ac567157fc56fa3fabafa5ad9a95bcadb512da8011a2c57dc WatchSource:0}: Error finding container 8af4eb0084ecd75ac567157fc56fa3fabafa5ad9a95bcadb512da8011a2c57dc: Status 404 returned error can't find the container with id 8af4eb0084ecd75ac567157fc56fa3fabafa5ad9a95bcadb512da8011a2c57dc
Nov 01 09:22:23.444702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.444667    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:23.444813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.444730    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:23.444813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.444762    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:23.785718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.785681    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:23.790961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:23.790929    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:24.191626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:24.191491    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" event={"ID":"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f","Type":"ContainerStarted","Data":"c627ed2e39bbd069c6dc5f7212c62ddb4c60a04a7ab419adcf82f24b7de0eedd"}
Nov 01 09:22:24.191626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:24.191621    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" event={"ID":"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f","Type":"ContainerStarted","Data":"8af4eb0084ecd75ac567157fc56fa3fabafa5ad9a95bcadb512da8011a2c57dc"}
Nov 01 09:22:24.191992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:24.191970    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:24.192994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:24.192974    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:22:24.304436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:24.304376    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podStartSLOduration=2.304360554 podStartE2EDuration="2.304360554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:24.303217169 +0000 UTC m=+3017.497611723" watchObservedRunningTime="2026-11-01 09:22:24.304360554 +0000 UTC m=+3017.498755110"
Nov 01 09:22:24.929274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:24.929239    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" path="/var/lib/kubelet/pods/23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0/volumes"
Nov 01 09:22:25.011037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:25.010995    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:25.016713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:25.016683    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:25.198771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:25.198680    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:25.199652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:25.199628    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:22:26.179570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:26.179424    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:26.384054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:26.384022    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:26.554072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:26.554003    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:27.137893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:27.137864    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:27.532684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:27.532652    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:27.712766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:27.712733    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:27.882696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:27.866052    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:28.105273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:28.105236    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"]
Nov 01 09:22:28.151463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:28.151381    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aaec97f-bpj8x"]
Nov 01 09:22:28.935758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:28.935720    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcf6b0d-0549-4dd3-a057-5d9411b63883" path="/var/lib/kubelet/pods/bdcf6b0d-0549-4dd3-a057-5d9411b63883/volumes"
Nov 01 09:22:30.142416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.142379    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:30.327066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.327031    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:30.557732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.557698    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"]
Nov 01 09:22:30.558261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.558247    4314 cpu_manager.go:414] "containerMap: removing container" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy"
Nov 01 09:22:30.558308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.558262    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy"
Nov 01 09:22:30.558308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.558272    4314 cpu_manager.go:414] "containerMap: removing container" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy"
Nov 01 09:22:30.558308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.558278    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy"
Nov 01 09:22:30.558419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.558393    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy"
Nov 01 09:22:30.558419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.558403    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d7ab31-8cf7-4f36-b7b4-5925c6fd90e0" containerName="istio-proxy"
Nov 01 09:22:30.563299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.563276    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.616236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.616199    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"]
Nov 01 09:22:30.651558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.648680    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.651558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.648768    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qhv\" (UniqueName: \"kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv\") pod \"payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.659807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.658146    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-58de5a\""
Nov 01 09:22:30.659807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.658598    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-58de5a-dockercfg-x67l9\""
Nov 01 09:22:30.762102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.758459    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.762102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.758637    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qhv\" (UniqueName: \"kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv\") pod \"payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.762102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.760108    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.797714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.796602    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"]
Nov 01 09:22:30.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.802382    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:30.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.803459    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"]
Nov 01 09:22:30.816938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.816908    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qhv\" (UniqueName: \"kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv\") pod \"payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.879143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.879110    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:30.967806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.967763    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:30.968034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:30.967857    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59zs\" (UniqueName: \"kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs\") pod \"payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:31.068816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.068736    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:31.068816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.068794    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59zs\" (UniqueName: \"kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs\") pod \"payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:31.069518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.069499    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:31.126470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.126431    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59zs\" (UniqueName: \"kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs\") pod \"payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:31.260626 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:22:31.260573    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2122f13b_f67a_43bc_abcf_7e434a8b2eff.slice/crio-0a9fd2edc2ee087d682c70e3950ecdfe764e267a8716b2db0257452962f9bf1f WatchSource:0}: Error finding container 0a9fd2edc2ee087d682c70e3950ecdfe764e267a8716b2db0257452962f9bf1f: Status 404 returned error can't find the container with id 0a9fd2edc2ee087d682c70e3950ecdfe764e267a8716b2db0257452962f9bf1f
Nov 01 09:22:31.261087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.260649    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"]
Nov 01 09:22:31.422268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.422237    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:31.794774 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:22:31.794739    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb485d1df_e0e6_46a9_87b2_6aa46a71c488.slice/crio-aa92ee49e07d228192f803eed65f7d0d51eda348cee9849020264e69d3f0f05a WatchSource:0}: Error finding container aa92ee49e07d228192f803eed65f7d0d51eda348cee9849020264e69d3f0f05a: Status 404 returned error can't find the container with id aa92ee49e07d228192f803eed65f7d0d51eda348cee9849020264e69d3f0f05a
Nov 01 09:22:31.795246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:31.795217    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"]
Nov 01 09:22:32.254319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.254279    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" event={"ID":"2122f13b-f67a-43bc-abcf-7e434a8b2eff","Type":"ContainerStarted","Data":"471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14"}
Nov 01 09:22:32.254524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.254344    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" event={"ID":"2122f13b-f67a-43bc-abcf-7e434a8b2eff","Type":"ContainerStarted","Data":"0a9fd2edc2ee087d682c70e3950ecdfe764e267a8716b2db0257452962f9bf1f"}
Nov 01 09:22:32.254524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.254378    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:32.256264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.256230    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" event={"ID":"b485d1df-e0e6-46a9-87b2-6aa46a71c488","Type":"ContainerStarted","Data":"3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194"}
Nov 01 09:22:32.256396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.256283    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" event={"ID":"b485d1df-e0e6-46a9-87b2-6aa46a71c488","Type":"ContainerStarted","Data":"aa92ee49e07d228192f803eed65f7d0d51eda348cee9849020264e69d3f0f05a"}
Nov 01 09:22:32.256396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.256375    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:32.369284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.369229    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" podStartSLOduration=2.369213578 podStartE2EDuration="2.369213578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:32.366948808 +0000 UTC m=+3025.561343364" watchObservedRunningTime="2026-11-01 09:22:32.369213578 +0000 UTC m=+3025.563608132"
Nov 01 09:22:32.486749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.486676    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" podStartSLOduration=2.486657008 podStartE2EDuration="2.486657008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:32.476473181 +0000 UTC m=+3025.670867739" watchObservedRunningTime="2026-11-01 09:22:32.486657008 +0000 UTC m=+3025.681051560"
Nov 01 09:22:32.514930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.514854    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:32.655398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:32.655333    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:33.062750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.062713    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"]
Nov 01 09:22:33.069562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.068987    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.113423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.113391    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"]
Nov 01 09:22:33.117981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.117941    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-e40e82-dockercfg-v8zvh\""
Nov 01 09:22:33.118147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.117941    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-e40e82\""
Nov 01 09:22:33.211666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.211635    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") " pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.211850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.211681    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqh28\" (UniqueName: \"kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28\") pod \"payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") " pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.265956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.265927    4314 generic.go:356] "Generic (PLEG): container finished" podID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerID="9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c" exitCode=0
Nov 01 09:22:33.266125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.266001    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" event={"ID":"94455c39-0ea2-43f4-861d-97e6bc7bfca3","Type":"ContainerDied","Data":"9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"}
Nov 01 09:22:33.266502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.266478    4314 scope.go:122] "RemoveContainer" containerID="9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"
Nov 01 09:22:33.268239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.268206    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:33.268331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.268273    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:33.268331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.268315    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:33.313607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.313512    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") " pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.313745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.313604    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh28\" (UniqueName: \"kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28\") pod \"payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") " pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.314405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.314380    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") " pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.371081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.371054    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqh28\" (UniqueName: \"kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28\") pod \"payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") " pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.385489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.385441    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:33.779050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:33.779025    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"]
Nov 01 09:22:33.779740 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:22:33.779710    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017f1b6e_123d_49ae_a8cc_c48bfbfa83d9.slice/crio-bf32bd2068051b146c6216acc3245993b7c51e8ac7dce446923919fb9442b74f WatchSource:0}: Error finding container bf32bd2068051b146c6216acc3245993b7c51e8ac7dce446923919fb9442b74f: Status 404 returned error can't find the container with id bf32bd2068051b146c6216acc3245993b7c51e8ac7dce446923919fb9442b74f
Nov 01 09:22:34.278412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.278375    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" event={"ID":"94455c39-0ea2-43f4-861d-97e6bc7bfca3","Type":"ContainerStarted","Data":"421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd"}
Nov 01 09:22:34.280913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.280883    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj" event={"ID":"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9","Type":"ContainerStarted","Data":"7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d"}
Nov 01 09:22:34.281041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.280938    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj" event={"ID":"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9","Type":"ContainerStarted","Data":"bf32bd2068051b146c6216acc3245993b7c51e8ac7dce446923919fb9442b74f"}
Nov 01 09:22:34.281041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.280964    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:34.549966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.549857    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj" podStartSLOduration=2.549842259 podStartE2EDuration="2.549842259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:34.545590219 +0000 UTC m=+3027.739984767" watchObservedRunningTime="2026-11-01 09:22:34.549842259 +0000 UTC m=+3027.744236813"
Nov 01 09:22:34.806157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.806059    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:22:34.806318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.806271    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 01 09:22:34.806363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:34.806322    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 01 09:22:35.806145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:35.806103    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 01 09:22:35.806685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:35.806167    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 01 09:22:36.806129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:36.806091    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 01 09:22:36.806319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:36.806149    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 01 09:22:37.725213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:37.725182    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:37.806578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:37.806046    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 01 09:22:37.806578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:37.806113    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 01 09:22:38.805916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:38.805881    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 01 09:22:38.806107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:38.805947    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 01 09:22:39.806757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:39.805945    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 01 09:22:39.806757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:39.806035    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 01 09:22:40.810233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:40.810204    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:22:40.810645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:40.810633    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:22:40.811348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:40.811327    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:22:41.324881 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:22:41.324848    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:22:43.026351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:43.026319    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:43.047330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:43.047302    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"]
Nov 01 09:22:43.094741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:43.094612    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe12058-s8c49"]
Nov 01 09:22:43.283959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:43.283871    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:22:43.284136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:43.284040    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:22:44.929130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:44.929096    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176114b6-8039-496b-9435-6b83d642bd64" path="/var/lib/kubelet/pods/176114b6-8039-496b-9435-6b83d642bd64/volumes"
Nov 01 09:22:45.294642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:45.294614    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:22:48.001262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:48.001220    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:49.468394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:49.468363    4314 generic.go:356] "Generic (PLEG): container finished" podID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerID="6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5" exitCode=0
Nov 01 09:22:49.468961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:49.468443    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" event={"ID":"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1","Type":"ContainerDied","Data":"6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"}
Nov 01 09:22:49.469037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:49.469006    4314 scope.go:122] "RemoveContainer" containerID="6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"
Nov 01 09:22:49.470670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:49.470643    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:49.470740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:49.470715    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:49.470778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:49.470751    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:50.489424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:50.489386    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" event={"ID":"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1","Type":"ContainerStarted","Data":"7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4"}
Nov 01 09:22:50.676712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:50.676674    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:22:51.382556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:51.382207    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:22:51.382556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:51.382437    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 01 09:22:51.382556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:51.382486    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 01 09:22:52.048580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:52.048549    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"]
Nov 01 09:22:52.094625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:52.094591    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c843d5ae-xpqt8"]
Nov 01 09:22:52.383060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:52.382967    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 01 09:22:52.383060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:52.383024    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 01 09:22:52.794678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:52.794641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:22:52.929922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:52.929885    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575decce-06ba-40bf-9e56-db6982769f37" path="/var/lib/kubelet/pods/575decce-06ba-40bf-9e56-db6982769f37/volumes"
Nov 01 09:22:53.382907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:53.382871    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 01 09:22:53.383352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:53.382926    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 01 09:22:54.383028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.382991    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 01 09:22:54.383685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.383647    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 01 09:22:54.904805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.904771    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"]
Nov 01 09:22:54.910204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.910181    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.979774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.979736    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z5ppc\""
Nov 01 09:22:54.980089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.980070    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 09:22:54.980250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.980236    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 09:22:54.980440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.980424    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 09:22:54.985415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985383    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.985575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985509    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.985659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985591    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.985659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985641    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.985793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985668    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.985793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985683    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qq4r\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:54.985915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:54.985830    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.029154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.029125    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"]
Nov 01 09:22:55.087701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.087661    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.087947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.087736    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.087947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.087777    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.087947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.087801    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qq4r\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.087947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.087858    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.088173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.087998    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.088173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.088126    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.088850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.088822    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.090357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.090326    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.090467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.090417    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.090659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.090641    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.090708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.090680    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.184654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.184484    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qq4r\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.187467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.187298    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dqd8g\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.222953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.222924    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:55.380757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.380720    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Nov 01 09:22:55.381105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.381044    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Nov 01 09:22:55.669843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.669803    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"]
Nov 01 09:22:55.701609 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:22:55.701579    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afc937e_7f73_421c_8740_d7eafcfe12f1.slice/crio-c7c0de83101a04687100d0a5b6200d613b7a6f6196b94504ce3bd7d7c931de2a WatchSource:0}: Error finding container c7c0de83101a04687100d0a5b6200d613b7a6f6196b94504ce3bd7d7c931de2a: Status 404 returned error can't find the container with id c7c0de83101a04687100d0a5b6200d613b7a6f6196b94504ce3bd7d7c931de2a
Nov 01 09:22:55.704284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.704248    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:55.704411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:55.704323    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:22:56.382847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.382807    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:22:56.383996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.383978    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:22:56.536758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.536719    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" event={"ID":"9afc937e-7f73-421c-8740-d7eafcfe12f1","Type":"ContainerStarted","Data":"d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6"}
Nov 01 09:22:56.536917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.536768    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" event={"ID":"9afc937e-7f73-421c-8740-d7eafcfe12f1","Type":"ContainerStarted","Data":"c7c0de83101a04687100d0a5b6200d613b7a6f6196b94504ce3bd7d7c931de2a"}
Nov 01 09:22:56.537000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.536980    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:56.538849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.538827    4314 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dqd8g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:22:56.538918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.538875    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:22:56.658264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:56.658177    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" podStartSLOduration=2.658162427 podStartE2EDuration="2.658162427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:56.653438479 +0000 UTC m=+3049.847833034" watchObservedRunningTime="2026-11-01 09:22:56.658162427 +0000 UTC m=+3049.852556981"
Nov 01 09:22:57.544690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:57.544656    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:22:57.590675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:22:57.590641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:02.391751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:02.391718    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:05.048043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:05.048002    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"]
Nov 01 09:23:05.094154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:05.094118    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ee64da6-57cgj"]
Nov 01 09:23:05.757883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:05.757835    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:06.952221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:06.952180    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c447073d-aeb9-4b4c-bd83-d569030fd9e4" path="/var/lib/kubelet/pods/c447073d-aeb9-4b4c-bd83-d569030fd9e4/volumes"
Nov 01 09:23:07.699877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:07.699841    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:08.046115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.046077    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"]
Nov 01 09:23:08.092198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.092164    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff175c7-x9gd4"]
Nov 01 09:23:08.732455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.732416    4314 scope.go:122] "RemoveContainer" containerID="c6e1c9a990e22d60060c4349e87027ef1acbd90d9083f90bb143609fa20ad197"
Nov 01 09:23:08.743079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.743052    4314 scope.go:122] "RemoveContainer" containerID="13d8b812f4627cffd6b0f2a0424aa5398331933785ee295b80efbcec71122b05"
Nov 01 09:23:08.753008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.752987    4314 scope.go:122] "RemoveContainer" containerID="d0240e8dba5ddb062bdc1a6498cd749f5d93c35a0c41bb5f51ddeabe9f7d5c19"
Nov 01 09:23:08.762796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.762772    4314 scope.go:122] "RemoveContainer" containerID="f337150745fccac2996feebe2f6dfcf40da938a443239aafae990b929921decb"
Nov 01 09:23:08.774401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.774374    4314 scope.go:122] "RemoveContainer" containerID="36a1a4ab8ab50558f8be1e54c393ba38bd5b07a0f500927001566ee030170e98"
Nov 01 09:23:08.783298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.783278    4314 scope.go:122] "RemoveContainer" containerID="fffa5f92190761db7222c110f3f2864acec896e2de0e7a086a8ee645d4c948df"
Nov 01 09:23:08.792734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.792717    4314 scope.go:122] "RemoveContainer" containerID="655124e121d2c24269bc596805b2aa136d18ca12f0715ac9b2956575b30f71d8"
Nov 01 09:23:08.801219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.801202    4314 scope.go:122] "RemoveContainer" containerID="0bb47b9ea1b5de09abe69a36fbf0210055a7665b959de849fc35f60a06448145"
Nov 01 09:23:08.929728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:08.929692    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767a0a5e-5d17-48d0-a141-70025df7e887" path="/var/lib/kubelet/pods/767a0a5e-5d17-48d0-a141-70025df7e887/volumes"
Nov 01 09:23:09.074242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:09.074156    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:09.397994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:09.397916    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:11.325751 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:23:11.325713    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:23:12.893669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:12.893616    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:17.486590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:17.486555    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:18.135459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:18.135424    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:21.046847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:21.046812    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"]
Nov 01 09:23:21.061377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:21.061339    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69b7da9-4t5bp"]
Nov 01 09:23:22.929672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:22.929636    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f" path="/var/lib/kubelet/pods/984feac5-a47f-4a8a-b5eb-cfd7f3ae6d3f/volumes"
Nov 01 09:23:23.096041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:23.096005    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:27.838125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:27.838083    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:28.002411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:28.002338    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:28.045792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:28.045753    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"]
Nov 01 09:23:28.093303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:28.093212    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e8be07-b9tl4"]
Nov 01 09:23:28.295466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:28.295438    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:28.929733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:28.929703    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a752e6e-ea64-459e-adc8-6a32e82a73be" path="/var/lib/kubelet/pods/4a752e6e-ea64-459e-adc8-6a32e82a73be/volumes"
Nov 01 09:23:29.065134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:29.065102    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"]
Nov 01 09:23:29.110907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:29.110869    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3b093c-f4q7c"]
Nov 01 09:23:30.137594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:30.137561    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:30.938706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:30.938664    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa71b344-3b51-4a78-8c55-fb9a28829eac" path="/var/lib/kubelet/pods/aa71b344-3b51-4a78-8c55-fb9a28829eac/volumes"
Nov 01 09:23:33.028955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:33.028912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:33.189800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:33.189746    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:33.337456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:33.337345    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:36.047917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:36.047879    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"]
Nov 01 09:23:36.093457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:36.093419    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ecd3c9f-bkgsc"]
Nov 01 09:23:36.929675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:36.929639    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1" path="/var/lib/kubelet/pods/48ad0cd3-ba79-49ed-8b6c-041ab8b3aba1/volumes"
Nov 01 09:23:37.976446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:37.976394    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:39.047357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:39.047320    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"]
Nov 01 09:23:39.093063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:39.093016    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0fce89-scwgx"]
Nov 01 09:23:40.929823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:40.929787    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175abac6-0887-435d-a53f-929b2cafa367" path="/var/lib/kubelet/pods/175abac6-0887-435d-a53f-929b2cafa367/volumes"
Nov 01 09:23:41.324442 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:23:41.324400    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:23:42.816224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:42.816180    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:47.522605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:47.522569    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:52.415644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.415603    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:52.981892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.981859    4314 generic.go:356] "Generic (PLEG): container finished" podID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerID="43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce" exitCode=0
Nov 01 09:23:52.982152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.981936    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" event={"ID":"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5","Type":"ContainerDied","Data":"43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"}
Nov 01 09:23:52.982483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.982465    4314 scope.go:122] "RemoveContainer" containerID="43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"
Nov 01 09:23:52.984200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.984169    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:23:52.984304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.984243    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:23:52.984304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:52.984286    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:23:53.991709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:53.991677    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" event={"ID":"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5","Type":"ContainerStarted","Data":"334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f"}
Nov 01 09:23:54.192486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:54.192448    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:23:54.782817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:54.782762    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:23:54.784120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:54.784083    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 01 09:23:54.784264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:54.784141    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 01 09:23:55.783104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:55.783053    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 01 09:23:55.783685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:55.783115    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 01 09:23:56.783044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:56.783000    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 01 09:23:56.783270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:56.783070    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 01 09:23:57.047691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:57.047596    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"]
Nov 01 09:23:57.093246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:57.093207    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f85aea-l7gxn"]
Nov 01 09:23:57.217224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:57.217191    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:23:57.783107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:57.783066    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 01 09:23:57.783307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:57.783141    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 01 09:23:58.782991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:58.782952    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 01 09:23:58.783221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:58.783019    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 01 09:23:58.929608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:58.929567    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ca9908-15df-4b73-90da-2c4d9b0bb4af" path="/var/lib/kubelet/pods/c4ca9908-15df-4b73-90da-2c4d9b0bb4af/volumes"
Nov 01 09:23:59.783106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:59.783058    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 01 09:23:59.783439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:23:59.783116    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 01 09:24:00.787638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:00.787598    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:24:00.788738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:00.788715    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:24:00.792147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:00.792123    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:02.001072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:02.001032    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:06.795374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:06.795341    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:08.868339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:08.868290    4314 scope.go:122] "RemoveContainer" containerID="72975d332216b4c65df802c1b8105fae4d125dae401be556678dca241aedbff6"
Nov 01 09:24:08.880948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:08.880925    4314 scope.go:122] "RemoveContainer" containerID="b5e29ae580e28808bf477fff03ac2a472f7d7a5b1286fa0a9edf421375638f13"
Nov 01 09:24:08.893200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:08.893175    4314 scope.go:122] "RemoveContainer" containerID="c2e439b3ee754b22e8bc891a83a6dd9138b45f12f1b01fd98c094713ed2121c7"
Nov 01 09:24:08.904414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:08.904388    4314 scope.go:122] "RemoveContainer" containerID="5201486697d7910db864664448fea4ce5eb741e24daeee3216228e108f7d3796"
Nov 01 09:24:08.917982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:08.917842    4314 scope.go:122] "RemoveContainer" containerID="94bd5557b922c70ddc876cb6d0a8bd949bb5b1a81dc681e1d05aaee590b4afae"
Nov 01 09:24:10.323411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:10.321482    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": read tcp 10.132.0.2:52618->10.132.0.226:15021: read: connection reset by peer" start-of-body=
Nov 01 09:24:10.323411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:10.321589    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": read tcp 10.132.0.2:52618->10.132.0.226:15021: read: connection reset by peer"
Nov 01 09:24:10.684334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:10.684237    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:11.157164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.157116    4314 generic.go:356] "Generic (PLEG): container finished" podID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerID="c627ed2e39bbd069c6dc5f7212c62ddb4c60a04a7ab419adcf82f24b7de0eedd" exitCode=0
Nov 01 09:24:11.157356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.157188    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" event={"ID":"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f","Type":"ContainerDied","Data":"c627ed2e39bbd069c6dc5f7212c62ddb4c60a04a7ab419adcf82f24b7de0eedd"}
Nov 01 09:24:11.157796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.157777    4314 scope.go:122] "RemoveContainer" containerID="c627ed2e39bbd069c6dc5f7212c62ddb4c60a04a7ab419adcf82f24b7de0eedd"
Nov 01 09:24:11.159499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.159465    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:24:11.159603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.159553    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:24:11.159603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.159590    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:24:11.330272 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:24:11.330237    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:24:11.786796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:11.786754    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:12.166479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:12.166382    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" event={"ID":"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f","Type":"ContainerStarted","Data":"7e7878e99d50e2bd231f3e822d50f73519a96c98b791935b38a97f281271e33b"}
Nov 01 09:24:13.011167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:13.011130    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:24:13.011741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:13.011392    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 01 09:24:13.011741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:13.011449    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 01 09:24:14.012023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:14.011987    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 01 09:24:14.012462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:14.012041    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 01 09:24:14.956384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:14.956349    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:15.012057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:15.012015    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 01 09:24:15.012593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:15.012093    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 01 09:24:16.011233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:16.011191    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 01 09:24:16.011458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:16.011272    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 01 09:24:16.597036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:16.596963    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:17.011650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:17.011617    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 01 09:24:17.011842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:17.011677    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 01 09:24:17.815847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:17.815806    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:18.011468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:18.011436    4314 patch_prober.go:29] interesting pod/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 01 09:24:18.011710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:18.011497    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 01 09:24:19.015183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:19.015145    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:24:19.015661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:19.015637    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:24:19.016408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:19.016389    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:24:21.373493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:21.373453    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:26.144773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:26.144739    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:30.987628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:30.987591    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:31.518218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:31.518183    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"]
Nov 01 09:24:31.518427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:31.518406    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy" containerID="cri-o://20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354" gracePeriod=30
Nov 01 09:24:35.786375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:35.786338    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:36.776971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.776946    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:24:36.871348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871272    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871330    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871388    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871441    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871487    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871646    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871659    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871729    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871766    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871798    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871816    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token\") pod \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\" (UID: \"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d\") "
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871815    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:24:36.871909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.871886    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:24:36.872507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.872005    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:24:36.872635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.872613    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.872682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.872643    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.872682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.872659    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.872682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.872674    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.872797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.872688    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.873796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.873770    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "kube-api-access-cptx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:24:36.873909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.873890    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:24:36.874049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.874027    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:24:36.874142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.874094    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy" pod "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" (UID: "2bcfb7cd-a61b-476b-bc98-5ea9bba3447d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:24:36.973943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.973915    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.973943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.973942    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.974180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.973959    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cptx6\" (UniqueName: \"kubernetes.io/projected/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-kube-api-access-cptx6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:36.974180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:36.973974    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:24:37.367037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.367005    4314 generic.go:356] "Generic (PLEG): container finished" podID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerID="20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354" exitCode=0
Nov 01 09:24:37.367245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.367061    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"
Nov 01 09:24:37.367245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.367094    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" event={"ID":"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d","Type":"ContainerDied","Data":"20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354"}
Nov 01 09:24:37.367245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.367159    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785" event={"ID":"2bcfb7cd-a61b-476b-bc98-5ea9bba3447d","Type":"ContainerDied","Data":"4097d4875fbb5e058d3eaf8f2843bc370f48c1e3e86f57b69989ca9990f87c83"}
Nov 01 09:24:37.367245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.367192    4314 scope.go:122] "RemoveContainer" containerID="20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354"
Nov 01 09:24:37.377190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.377175    4314 scope.go:122] "RemoveContainer" containerID="20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354"
Nov 01 09:24:37.377466 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:24:37.377445    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354\": container with ID starting with 20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354 not found: ID does not exist" containerID="20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354"
Nov 01 09:24:37.377522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.377475    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354"} err="failed to get container status \"20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354\": rpc error: code = NotFound desc = could not find container \"20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354\": container with ID starting with 20b691377db714a6e440c5093fb4b28e05aeb4b3229f7dce089cc5eba6e3f354 not found: ID does not exist"
Nov 01 09:24:37.568604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.568574    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"]
Nov 01 09:24:37.614853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:37.614824    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-tv785"]
Nov 01 09:24:38.930114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:38.930079    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" path="/var/lib/kubelet/pods/2bcfb7cd-a61b-476b-bc98-5ea9bba3447d/volumes"
Nov 01 09:24:40.581912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:40.581859    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:41.326015 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:24:41.325977    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:24:43.915793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:43.915760    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:44.342672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:44.342639    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:44.494278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:44.494233    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:44.923236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:44.923202    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:45.192724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:45.192641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:45.898255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:45.898206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:47.736035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:47.736004    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:49.835365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:49.835333    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:50.038426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:50.038387    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:50.228752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:50.228662    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:51.297301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:51.297258    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:53.002741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.002695    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:53.173011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.172976    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:24:53.404440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.404352    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"]
Nov 01 09:24:53.405070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.405047    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy"
Nov 01 09:24:53.405198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.405071    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy"
Nov 01 09:24:53.405258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.405208    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcfb7cd-a61b-476b-bc98-5ea9bba3447d" containerName="istio-proxy"
Nov 01 09:24:53.408867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.408846    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.458139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.458086    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a83714\""
Nov 01 09:24:53.491747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.491701    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp\") pod \"payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.491942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.491809    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.535214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.535177    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a83714-dockercfg-gv6zm\""
Nov 01 09:24:53.550669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.550633    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"]
Nov 01 09:24:53.599195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.599136    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.599761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.599733    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp\") pod \"payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.600989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.600966    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.656855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.656778    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp\") pod \"payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:53.727805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:53.727779    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:54.094983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:54.094951    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"]
Nov 01 09:24:54.095728 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:24:54.095703    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56ad473_a2a7_4da5_87ed_1929a0ee715b.slice/crio-caf103ca0e4adb3b45dec7933bf7073321799b3de21e564f1cc2f99217ce97b0 WatchSource:0}: Error finding container caf103ca0e4adb3b45dec7933bf7073321799b3de21e564f1cc2f99217ce97b0: Status 404 returned error can't find the container with id caf103ca0e4adb3b45dec7933bf7073321799b3de21e564f1cc2f99217ce97b0
Nov 01 09:24:54.486046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:54.486011    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87" event={"ID":"d56ad473-a2a7-4da5-87ed-1929a0ee715b","Type":"ContainerStarted","Data":"985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d"}
Nov 01 09:24:54.486198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:54.486067    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87" event={"ID":"d56ad473-a2a7-4da5-87ed-1929a0ee715b","Type":"ContainerStarted","Data":"caf103ca0e4adb3b45dec7933bf7073321799b3de21e564f1cc2f99217ce97b0"}
Nov 01 09:24:54.486198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:54.486097    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:24:54.597617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:54.597516    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87" podStartSLOduration=1.597498911 podStartE2EDuration="1.597498911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:24:54.593845952 +0000 UTC m=+3167.788240507" watchObservedRunningTime="2026-11-01 09:24:54.597498911 +0000 UTC m=+3167.791893467"
Nov 01 09:24:55.799374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.799340    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"]
Nov 01 09:24:55.799781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.799553    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy" containerID="cri-o://8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b" gracePeriod=30
Nov 01 09:24:55.846487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.846458    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"]
Nov 01 09:24:55.846716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.846692    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" containerID="cri-o://2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215" gracePeriod=30
Nov 01 09:24:55.850296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.850273    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"]
Nov 01 09:24:55.850478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.850457    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" containerID="cri-o://bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d" gracePeriod=30
Nov 01 09:24:55.891751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.891723    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"]
Nov 01 09:24:55.891903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.891887    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" containerID="cri-o://33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934" gracePeriod=30
Nov 01 09:24:55.966269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.966236    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"]
Nov 01 09:24:55.966490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.966464    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy" containerID="cri-o://ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab" gracePeriod=30
Nov 01 09:24:55.978611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.978586    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"]
Nov 01 09:24:55.978795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:55.978772    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy" containerID="cri-o://7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073" gracePeriod=30
Nov 01 09:24:56.003456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:56.003427    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"]
Nov 01 09:24:56.003673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:56.003649    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy" containerID="cri-o://71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686" gracePeriod=30
Nov 01 09:24:56.008018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:56.007991    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"]
Nov 01 09:24:56.008227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:56.008209    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy" containerID="cri-o://78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254" gracePeriod=30
Nov 01 09:24:56.210527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:24:56.210442    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:00.846920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:00.846891    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:25:00.847293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:00.846951    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:25:00.952929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:00.952904    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 09:25:00.953054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:00.952954    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 09:25:01.099016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.098748    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:01.262405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.262374    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:25:01.265852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.265828    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:25:01.290723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.290696    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:25:01.380836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.380803    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:25:01.421071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421045    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.421292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421096    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.421292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421127    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.421292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421164    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.421519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421432    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.421619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421567    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.421820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421800    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.421909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421848    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.421909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421880    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.422031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421918    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.422031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421963    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.422031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.421997    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.422201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422045    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.422201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422078    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.422201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422124    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.422201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422171    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.422414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422228    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.422414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422267    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.422414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422362    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.422414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422390    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.422414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422391    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.424954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.422450    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428194    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428241    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428284    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428361    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428436    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo\") pod \"eb42d12b-b56d-448b-8087-94a155a7b73d\" (UID: \"eb42d12b-b56d-448b-8087-94a155a7b73d\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428518    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert\") pod \"51883dec-a44f-4fa9-9032-b831d3551b60\" (UID: \"51883dec-a44f-4fa9-9032-b831d3551b60\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428575    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.428642    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket\") pod \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\" (UID: \"bb6f95e2-4dca-4643-a8b2-832293ad7d86\") "
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.429335    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42d12b-b56d-448b-8087-94a155a7b73d-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.429357    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.429555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.429370    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.432106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.423372    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.432106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.423375    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.432106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.423596    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.432106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.430870    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.432106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.431173    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.432106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.431824    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.433613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.432812    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.434624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.432897    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.434624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.433139    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.434624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.433336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.434624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.433334    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.435471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.435119    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.436193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.436169    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.436777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.436310    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "kube-api-access-2nns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.436777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.436315    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.436777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.436359    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.437416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.437390    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.437612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.437589    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.438003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.437982    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.438173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.438151    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w" pod "eb42d12b-b56d-448b-8087-94a155a7b73d" (UID: "eb42d12b-b56d-448b-8087-94a155a7b73d"). InnerVolumeSpecName "kube-api-access-dfx8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.439092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.438453    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.439092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.438698    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp" pod "bb6f95e2-4dca-4643-a8b2-832293ad7d86" (UID: "bb6f95e2-4dca-4643-a8b2-832293ad7d86"). InnerVolumeSpecName "kube-api-access-xfnlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.439092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.438778    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.440693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.440667    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy" pod "51883dec-a44f-4fa9-9032-b831d3551b60" (UID: "51883dec-a44f-4fa9-9032-b831d3551b60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.441992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.441913    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:25:01.445987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.445965    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:25:01.450024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.450001    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 09:25:01.454979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.454962    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:25:01.530064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530027    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530074    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530127    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530169    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530196    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530281    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.530460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530337    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530371    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530376    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.530717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530476    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530518    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.530717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530553    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.530717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530526    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token\") pod \"8196af88-ac13-483a-8911-6a9bff599b3f\" (UID: \"8196af88-ac13-483a-8911-6a9bff599b3f\") "
Nov 01 09:25:01.530920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.530741    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.531153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531117    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531142    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531157    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfx8w\" (UniqueName: \"kubernetes.io/projected/eb42d12b-b56d-448b-8087-94a155a7b73d-kube-api-access-dfx8w\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531171    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531183    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531198    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531211    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531225    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8196af88-ac13-483a-8911-6a9bff599b3f-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531238    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nns6\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-kube-api-access-2nns6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531274    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531293    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531308    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531321    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51883dec-a44f-4fa9-9032-b831d3551b60-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531336    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531349    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531363    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531378    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnlp\" (UniqueName: \"kubernetes.io/projected/bb6f95e2-4dca-4643-a8b2-832293ad7d86-kube-api-access-xfnlp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531391    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51883dec-a44f-4fa9-9032-b831d3551b60-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531408    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531423    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42d12b-b56d-448b-8087-94a155a7b73d-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531437    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531450    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42d12b-b56d-448b-8087-94a155a7b73d-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531464    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531477    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51883dec-a44f-4fa9-9032-b831d3551b60-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531492    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531506    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531521    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531547    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6f95e2-4dca-4643-a8b2-832293ad7d86-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.531850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.531569    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51883dec-a44f-4fa9-9032-b831d3551b60-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.537118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.534899    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "kube-api-access-5bdff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.537118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.536973    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.537118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.537068    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.537118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.537073    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token" pod "8196af88-ac13-483a-8911-6a9bff599b3f" (UID: "8196af88-ac13-483a-8911-6a9bff599b3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.540888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.540866    4314 generic.go:356] "Generic (PLEG): container finished" podID="8196af88-ac13-483a-8911-6a9bff599b3f" containerID="33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934" exitCode=0
Nov 01 09:25:01.541069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.540917    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"
Nov 01 09:25:01.541069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.540941    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" event={"ID":"8196af88-ac13-483a-8911-6a9bff599b3f","Type":"ContainerDied","Data":"33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934"}
Nov 01 09:25:01.541069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.541024    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j" event={"ID":"8196af88-ac13-483a-8911-6a9bff599b3f","Type":"ContainerDied","Data":"2e10602e1fd4e38c921f8f2e0b17da29c4c3899ffb555357e137954da7dcd418"}
Nov 01 09:25:01.541069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.541058    4314 scope.go:122] "RemoveContainer" containerID="33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934"
Nov 01 09:25:01.542374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.542352    4314 generic.go:356] "Generic (PLEG): container finished" podID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerID="2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215" exitCode=0
Nov 01 09:25:01.542512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.542398    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"
Nov 01 09:25:01.542512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.542472    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" event={"ID":"bb6f95e2-4dca-4643-a8b2-832293ad7d86","Type":"ContainerDied","Data":"2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215"}
Nov 01 09:25:01.542683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.542514    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" event={"ID":"bb6f95e2-4dca-4643-a8b2-832293ad7d86","Type":"ContainerDied","Data":"5d52a05b52127cf01f2d98960a9c9a4ffa493449d4fd6d90496c7656084b2e85"}
Nov 01 09:25:01.544276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.544246    4314 generic.go:356] "Generic (PLEG): container finished" podID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerID="bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d" exitCode=0
Nov 01 09:25:01.544396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.544342    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" event={"ID":"eb42d12b-b56d-448b-8087-94a155a7b73d","Type":"ContainerDied","Data":"bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d"}
Nov 01 09:25:01.544396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.544383    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp" event={"ID":"eb42d12b-b56d-448b-8087-94a155a7b73d","Type":"ContainerDied","Data":"db1934b3391ea580f78077596a6f841c3d9f357de657b7c31ec2cfd4567ea0c7"}
Nov 01 09:25:01.544505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.544460    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"
Nov 01 09:25:01.550904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.550885    4314 generic.go:356] "Generic (PLEG): container finished" podID="51883dec-a44f-4fa9-9032-b831d3551b60" containerID="8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b" exitCode=0
Nov 01 09:25:01.551018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.550923    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"
Nov 01 09:25:01.551018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.550961    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" event={"ID":"51883dec-a44f-4fa9-9032-b831d3551b60","Type":"ContainerDied","Data":"8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b"}
Nov 01 09:25:01.551018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.551007    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w" event={"ID":"51883dec-a44f-4fa9-9032-b831d3551b60","Type":"ContainerDied","Data":"1decddf4cd1d7d83f74de0f1423c1f8eb1903ccf22b88afe60934d93831c7dc5"}
Nov 01 09:25:01.554397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.554376    4314 scope.go:122] "RemoveContainer" containerID="33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934"
Nov 01 09:25:01.556229 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.556172    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934\": container with ID starting with 33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934 not found: ID does not exist" containerID="33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934"
Nov 01 09:25:01.556229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.556205    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934"} err="failed to get container status \"33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934\": rpc error: code = NotFound desc = could not find container \"33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934\": container with ID starting with 33124b05f486c9f15ce7a7f17ec18d2dff26f66bf8dbba417314a2832082a934 not found: ID does not exist"
Nov 01 09:25:01.556229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.556227    4314 scope.go:122] "RemoveContainer" containerID="2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215"
Nov 01 09:25:01.558125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.558105    4314 generic.go:356] "Generic (PLEG): container finished" podID="16dae45d-623b-4f22-8039-36d33f892acf" containerID="71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686" exitCode=0
Nov 01 09:25:01.558413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.558148    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"
Nov 01 09:25:01.558724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.558162    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" event={"ID":"16dae45d-623b-4f22-8039-36d33f892acf","Type":"ContainerDied","Data":"71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686"}
Nov 01 09:25:01.558857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.558840    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb" event={"ID":"16dae45d-623b-4f22-8039-36d33f892acf","Type":"ContainerDied","Data":"3ddd976b87cd3249e6ea7b47c5cc57a10e9e4ebe2e1c3c6f149263da19c9282d"}
Nov 01 09:25:01.561187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.561168    4314 generic.go:356] "Generic (PLEG): container finished" podID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerID="ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab" exitCode=0
Nov 01 09:25:01.561284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.561187    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" event={"ID":"5b7b8005-f393-48e3-91f1-0d01bdc12bb9","Type":"ContainerDied","Data":"ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab"}
Nov 01 09:25:01.561284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.561232    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m" event={"ID":"5b7b8005-f393-48e3-91f1-0d01bdc12bb9","Type":"ContainerDied","Data":"b7f433096c6a647c3a7aefce261ef9f7f9f44c17f05e893764edc2c320f6557a"}
Nov 01 09:25:01.561284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.561235    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"
Nov 01 09:25:01.562845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.562826    4314 generic.go:356] "Generic (PLEG): container finished" podID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerID="7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073" exitCode=0
Nov 01 09:25:01.563325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.562906    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" event={"ID":"7e7c09be-8888-4b93-86b5-99de5c94fba3","Type":"ContainerDied","Data":"7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073"}
Nov 01 09:25:01.563325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.562955    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv" event={"ID":"7e7c09be-8888-4b93-86b5-99de5c94fba3","Type":"ContainerDied","Data":"ca9dc5c4ea7a1c207180d9cbf1840842d163388be05df543863ebc26a56b7c47"}
Nov 01 09:25:01.563558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.563515    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"
Nov 01 09:25:01.564681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.564662    4314 generic.go:356] "Generic (PLEG): container finished" podID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerID="78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254" exitCode=0
Nov 01 09:25:01.564806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.564690    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" event={"ID":"d617e251-78b9-44ee-b7a7-5996e5ede248","Type":"ContainerDied","Data":"78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254"}
Nov 01 09:25:01.564806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.564706    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"
Nov 01 09:25:01.564806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.564726    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q" event={"ID":"d617e251-78b9-44ee-b7a7-5996e5ede248","Type":"ContainerDied","Data":"2cb5dce291f321e19c336cf0e32d3d37fa63331510f940d2bce8a47c42397fe6"}
Nov 01 09:25:01.570904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.570883    4314 scope.go:122] "RemoveContainer" containerID="2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215"
Nov 01 09:25:01.571176 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.571156    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215\": container with ID starting with 2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215 not found: ID does not exist" containerID="2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215"
Nov 01 09:25:01.571317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.571184    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215"} err="failed to get container status \"2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215\": rpc error: code = NotFound desc = could not find container \"2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215\": container with ID starting with 2cf2274fc79cd2fad988345169f434702cbdcb6374efadf7d1acb80b0d7df215 not found: ID does not exist"
Nov 01 09:25:01.571317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.571208    4314 scope.go:122] "RemoveContainer" containerID="bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d"
Nov 01 09:25:01.580826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.580750    4314 scope.go:122] "RemoveContainer" containerID="bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d"
Nov 01 09:25:01.581059 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.581034    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d\": container with ID starting with bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d not found: ID does not exist" containerID="bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d"
Nov 01 09:25:01.581171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.581061    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d"} err="failed to get container status \"bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d\": rpc error: code = NotFound desc = could not find container \"bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d\": container with ID starting with bac3bd89c6948b6584196a9752df143931ec9f9042bffcd58f87c451bb59c20d not found: ID does not exist"
Nov 01 09:25:01.581171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.581077    4314 scope.go:122] "RemoveContainer" containerID="8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b"
Nov 01 09:25:01.592292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.592256    4314 scope.go:122] "RemoveContainer" containerID="8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b"
Nov 01 09:25:01.592625 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.592593    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b\": container with ID starting with 8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b not found: ID does not exist" containerID="8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b"
Nov 01 09:25:01.592733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.592632    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b"} err="failed to get container status \"8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b\": rpc error: code = NotFound desc = could not find container \"8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b\": container with ID starting with 8ee451d6464f4ab326e7cf0031478d5ff35d9f606f385f2808d129cb328fbd0b not found: ID does not exist"
Nov 01 09:25:01.592733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.592653    4314 scope.go:122] "RemoveContainer" containerID="71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686"
Nov 01 09:25:01.603952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.603934    4314 scope.go:122] "RemoveContainer" containerID="71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686"
Nov 01 09:25:01.604240 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.604222    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686\": container with ID starting with 71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686 not found: ID does not exist" containerID="71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686"
Nov 01 09:25:01.604299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.604245    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686"} err="failed to get container status \"71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686\": rpc error: code = NotFound desc = could not find container \"71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686\": container with ID starting with 71c1221ec5985f9a71dea05d394ab29ae1b68b2abbf03addab93c071ead0b686 not found: ID does not exist"
Nov 01 09:25:01.604299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.604261    4314 scope.go:122] "RemoveContainer" containerID="ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab"
Nov 01 09:25:01.615528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.615497    4314 scope.go:122] "RemoveContainer" containerID="ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab"
Nov 01 09:25:01.615836 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.615814    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab\": container with ID starting with ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab not found: ID does not exist" containerID="ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab"
Nov 01 09:25:01.615882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.615844    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab"} err="failed to get container status \"ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab\": rpc error: code = NotFound desc = could not find container \"ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab\": container with ID starting with ba431a9aed344f3b7cf32f00a3358346967a6cc44376e57fd16abe4fcbf79eab not found: ID does not exist"
Nov 01 09:25:01.615882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.615864    4314 scope.go:122] "RemoveContainer" containerID="7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073"
Nov 01 09:25:01.625214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.625197    4314 scope.go:122] "RemoveContainer" containerID="7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073"
Nov 01 09:25:01.625470 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.625455    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073\": container with ID starting with 7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073 not found: ID does not exist" containerID="7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073"
Nov 01 09:25:01.625517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.625475    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073"} err="failed to get container status \"7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073\": rpc error: code = NotFound desc = could not find container \"7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073\": container with ID starting with 7ec931458749b00d8ec75629fd09d8f41e8ff380bd5405aa8ef5c97cdcf3c073 not found: ID does not exist"
Nov 01 09:25:01.625517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.625489    4314 scope.go:122] "RemoveContainer" containerID="78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254"
Nov 01 09:25:01.631965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.631944    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.632078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.631986    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.632078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632009    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.632078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632042    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.632078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632070    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632100    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632137    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632167    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632236    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632286    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632289    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632324    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.632370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632330    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.632867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632381    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.632867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632412    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.632867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632452    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.632867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632501    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.632867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632550    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.633146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632783    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.633146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632889    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.633146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.632563    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.633146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633141    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.633368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633179    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.633368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633222    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.633368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633281    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.633368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633305    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.633368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633327    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.633368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633365    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.633711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633403    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.633711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633440    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.633711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633498    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.633711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633690    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.633923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633753    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.633923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633795    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.633923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633830    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.633923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633862    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket\") pod \"d617e251-78b9-44ee-b7a7-5996e5ede248\" (UID: \"d617e251-78b9-44ee-b7a7-5996e5ede248\") "
Nov 01 09:25:01.633923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633900    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.634308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633931    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5\") pod \"16dae45d-623b-4f22-8039-36d33f892acf\" (UID: \"16dae45d-623b-4f22-8039-36d33f892acf\") "
Nov 01 09:25:01.634308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.633961    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.634308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634017    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket\") pod \"7e7c09be-8888-4b93-86b5-99de5c94fba3\" (UID: \"7e7c09be-8888-4b93-86b5-99de5c94fba3\") "
Nov 01 09:25:01.634308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634053    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.634308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634084    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data\") pod \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\" (UID: \"5b7b8005-f393-48e3-91f1-0d01bdc12bb9\") "
Nov 01 09:25:01.634673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634311    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.634673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634609    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.634832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634683    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.634832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634781    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.634937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634883    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.634937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634905    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.634937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634922    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdff\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-kube-api-access-5bdff\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634937    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634952    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634966    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7c09be-8888-4b93-86b5-99de5c94fba3-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634980    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634996    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635013    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8196af88-ac13-483a-8911-6a9bff599b3f-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635027    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635041    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8196af88-ac13-483a-8911-6a9bff599b3f-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635055    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635069    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8196af88-ac13-483a-8911-6a9bff599b3f-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635083    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.635111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.634906    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.635773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635310    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.635773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635489    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.635773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635578    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.635953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635864    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.635953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.635877    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.636829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.636308    4314 scope.go:122] "RemoveContainer" containerID="78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254"
Nov 01 09:25:01.637123 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:01.637097    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254\": container with ID starting with 78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254 not found: ID does not exist" containerID="78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254"
Nov 01 09:25:01.637230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.637128    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254"} err="failed to get container status \"78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254\": rpc error: code = NotFound desc = could not find container \"78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254\": container with ID starting with 78848560841a99533ea9d1343c5d86e44e6b7e990af8ed8caffd560c97f46254 not found: ID does not exist"
Nov 01 09:25:01.637230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.637154    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.637398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.637370    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.637972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.637591    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.637972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.637754    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.637972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.637940    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:25:01.638328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.638302    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.639044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639017    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.639958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639804    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.639958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639806    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.639958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639897    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.639958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639912    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.639958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639923    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "kube-api-access-brnx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.640277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639966    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "kube-api-access-9jwgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.640277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.639986    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo" pod "7e7c09be-8888-4b93-86b5-99de5c94fba3" (UID: "7e7c09be-8888-4b93-86b5-99de5c94fba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.640277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.640053    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.640277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.640210    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "kube-api-access-8w54w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.640590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.640470    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy" pod "5b7b8005-f393-48e3-91f1-0d01bdc12bb9" (UID: "5b7b8005-f393-48e3-91f1-0d01bdc12bb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:25:01.640872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.640847    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.641193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.641172    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5" pod "16dae45d-623b-4f22-8039-36d33f892acf" (UID: "16dae45d-623b-4f22-8039-36d33f892acf"). InnerVolumeSpecName "kube-api-access-mj4v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:25:01.641675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.641656    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo" pod "d617e251-78b9-44ee-b7a7-5996e5ede248" (UID: "d617e251-78b9-44ee-b7a7-5996e5ede248"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:25:01.735865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735833    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jwgj\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-kube-api-access-9jwgj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.735865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735860    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.735865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735871    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735880    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnx8\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-kube-api-access-brnx8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735889    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735897    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735907    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae45d-623b-4f22-8039-36d33f892acf-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735915    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735922    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735932    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d617e251-78b9-44ee-b7a7-5996e5ede248-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735940    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735948    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735957    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735965    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735975    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w54w\" (UniqueName: \"kubernetes.io/projected/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-kube-api-access-8w54w\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735984    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d617e251-78b9-44ee-b7a7-5996e5ede248-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.735993    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae45d-623b-4f22-8039-36d33f892acf-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736002    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4v5\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-kube-api-access-mj4v5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736010    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7c09be-8888-4b93-86b5-99de5c94fba3-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736019    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7c09be-8888-4b93-86b5-99de5c94fba3-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736027    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736034    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736042    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae45d-623b-4f22-8039-36d33f892acf-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736050    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7b8005-f393-48e3-91f1-0d01bdc12bb9-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736059    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae45d-623b-4f22-8039-36d33f892acf-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:01.736096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:01.736067    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d617e251-78b9-44ee-b7a7-5996e5ede248-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:25:02.030520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.030486    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"]
Nov 01 09:25:02.036258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.036231    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-w2rlp"]
Nov 01 09:25:02.049126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.049101    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"]
Nov 01 09:25:02.052037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.052017    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14dea1ac-mh92b"]
Nov 01 09:25:02.143456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.140073    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"]
Nov 01 09:25:02.143456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.143407    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t"]
Nov 01 09:25:02.223466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.223438    4314 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:25:02.223615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.223484    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2nk4t" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:25:02.246326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.246298    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"]
Nov 01 09:25:02.251081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.251061    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2d55w"]
Nov 01 09:25:02.352426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.352390    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"]
Nov 01 09:25:02.361568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.361542    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4j88j"]
Nov 01 09:25:02.805495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.805460    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"]
Nov 01 09:25:02.810607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.810557    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rzl2q"]
Nov 01 09:25:02.911278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.911247    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"]
Nov 01 09:25:02.915542 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.915510    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-qjgzb"]
Nov 01 09:25:02.931300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.931276    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16dae45d-623b-4f22-8039-36d33f892acf" path="/var/lib/kubelet/pods/16dae45d-623b-4f22-8039-36d33f892acf/volumes"
Nov 01 09:25:02.931731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.931717    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252c5e54-29a5-4abd-b82d-54b426d3986b" path="/var/lib/kubelet/pods/252c5e54-29a5-4abd-b82d-54b426d3986b/volumes"
Nov 01 09:25:02.932148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.932134    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" path="/var/lib/kubelet/pods/51883dec-a44f-4fa9-9032-b831d3551b60/volumes"
Nov 01 09:25:02.932884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.932869    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" path="/var/lib/kubelet/pods/8196af88-ac13-483a-8911-6a9bff599b3f/volumes"
Nov 01 09:25:02.933286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.933271    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" path="/var/lib/kubelet/pods/bb6f95e2-4dca-4643-a8b2-832293ad7d86/volumes"
Nov 01 09:25:02.933882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.933867    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" path="/var/lib/kubelet/pods/d617e251-78b9-44ee-b7a7-5996e5ede248/volumes"
Nov 01 09:25:02.934557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:02.934526    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" path="/var/lib/kubelet/pods/eb42d12b-b56d-448b-8087-94a155a7b73d/volumes"
Nov 01 09:25:03.017163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:03.017140    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"]
Nov 01 09:25:03.023403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:03.023384    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-r8j5m"]
Nov 01 09:25:03.125894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:03.125823    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"]
Nov 01 09:25:03.129413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:03.129391    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-zptjv"]
Nov 01 09:25:04.928954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:04.928913    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" path="/var/lib/kubelet/pods/5b7b8005-f393-48e3-91f1-0d01bdc12bb9/volumes"
Nov 01 09:25:04.929592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:04.929571    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" path="/var/lib/kubelet/pods/7e7c09be-8888-4b93-86b5-99de5c94fba3/volumes"
Nov 01 09:25:05.496225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:05.496198    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:25:06.050508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:06.050473    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:08.991683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:08.991652    4314 scope.go:122] "RemoveContainer" containerID="c4998c199fea58b9c7dd91892263724d3e2313569a10e402000d437088126b8e"
Nov 01 09:25:09.004470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:09.004442    4314 scope.go:122] "RemoveContainer" containerID="30f6a1e284d884b5d26ba2e4513452e2a00c80d1ba66c79bc44af57ea1f731cc"
Nov 01 09:25:09.015438 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:09.015408    4314 scope.go:122] "RemoveContainer" containerID="29cf1f04664c508684469fbf1148fd5ad00f4c74c6befb96e5a163a0a2c9b862"
Nov 01 09:25:09.024850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:09.024823    4314 scope.go:122] "RemoveContainer" containerID="be681ab71d9762d3875ed171a54b0c5082885cb35af06a4cb6fc54e3b0436d31"
Nov 01 09:25:09.033687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:09.033656    4314 scope.go:122] "RemoveContainer" containerID="03fdaa8ef440df24f9645062465e73aa8236050f68b2af476fcc73aa37c03367"
Nov 01 09:25:09.042027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:09.042003    4314 scope.go:122] "RemoveContainer" containerID="8a0dd27d8290507dab89c4f251286fca5fcadb8ca1b07b6b5beeeacf39d5738b"
Nov 01 09:25:10.775667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:10.775633    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:11.324041 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:11.324004    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:25:15.527637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:15.527602    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:18.145964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:18.145934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:22.384099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:22.384023    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:25.206783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:25.206755    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:29.713330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713298    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"]
Nov 01 09:25:29.713832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713818    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713833    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713845    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713850    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713862    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713869    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713882    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy"
Nov 01 09:25:29.713888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713886    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713897    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713901    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713908    4314 cpu_manager.go:414] "containerMap: removing container" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713913    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713922    4314 cpu_manager.go:414] "containerMap: removing container" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713926    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713941    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.713946    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714021    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7b8005-f393-48e3-91f1-0d01bdc12bb9" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714027    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb42d12b-b56d-448b-8087-94a155a7b73d" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714035    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8196af88-ac13-483a-8911-6a9bff599b3f" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714043    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6f95e2-4dca-4643-a8b2-832293ad7d86" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714049    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7c09be-8888-4b93-86b5-99de5c94fba3" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714055    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="51883dec-a44f-4fa9-9032-b831d3551b60" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714061    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="16dae45d-623b-4f22-8039-36d33f892acf" containerName="istio-proxy"
Nov 01 09:25:29.714132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.714069    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d617e251-78b9-44ee-b7a7-5996e5ede248" containerName="istio-proxy"
Nov 01 09:25:29.719380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.719364    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.789353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.789320    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"openshift-service-ca.crt\""
Nov 01 09:25:29.814736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.814682    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:29.814889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.814801    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"kube-root-ca.crt\""
Nov 01 09:25:29.814889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.814803    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"default-dockercfg-qvwzz\""
Nov 01 09:25:29.814889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.814836    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"auth-test-model-65aeef-kserve-self-signed-certs\""
Nov 01 09:25:29.817902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.817882    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"]
Nov 01 09:25:29.836060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.836034    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.836198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.836101    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.836268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.836194    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.836268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.836232    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8q7x\" (UniqueName: \"kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.836394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.836358    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.836450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.836415    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937071    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937119    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937162    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937194    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937226    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937252    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8q7x\" (UniqueName: \"kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937581    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937630    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.937762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.937682    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.939297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.939275    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.939661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.939643    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.993992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.993964    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8q7x\" (UniqueName: \"kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x\") pod \"auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:29.994765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:29.994747    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:30.031619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:30.031597    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:30.401911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:30.401885    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"]
Nov 01 09:25:30.403008 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:25:30.402987    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e9d6bba_be30_4171_a6e9_56bf01d597dd.slice/crio-b8a849db7806d5426c24856aee61a339324cc28ef87c0b8d6313013e28021d5a WatchSource:0}: Error finding container b8a849db7806d5426c24856aee61a339324cc28ef87c0b8d6313013e28021d5a: Status 404 returned error can't find the container with id b8a849db7806d5426c24856aee61a339324cc28ef87c0b8d6313013e28021d5a
Nov 01 09:25:30.786819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:30.786784    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5" event={"ID":"5e9d6bba-be30-4171-a6e9-56bf01d597dd","Type":"ContainerStarted","Data":"74566c541c360491d05ff4acb226bc5207b4a153004550fea1e965a516ceadde"}
Nov 01 09:25:30.787202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:30.786834    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5" event={"ID":"5e9d6bba-be30-4171-a6e9-56bf01d597dd","Type":"ContainerStarted","Data":"b8a849db7806d5426c24856aee61a339324cc28ef87c0b8d6313013e28021d5a"}
Nov 01 09:25:30.899165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:30.899104    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5" podStartSLOduration=1.899090872 podStartE2EDuration="1.899090872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:25:30.895673791 +0000 UTC m=+3204.090068346" watchObservedRunningTime="2026-11-01 09:25:30.899090872 +0000 UTC m=+3204.093485426"
Nov 01 09:25:31.032370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:31.032337    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:33.619791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.619751    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"]
Nov 01 09:25:33.624031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.624007    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.716962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.716926    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:33.720045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.720027    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"openshift-service-ca.crt\""
Nov 01 09:25:33.720158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.720049    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"default-dockercfg-t5cr5\""
Nov 01 09:25:33.720430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.720408    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"auth-test-model-46bf99-kserve-self-signed-certs\""
Nov 01 09:25:33.720521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.720450    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"kube-root-ca.crt\""
Nov 01 09:25:33.723571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.723545    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"]
Nov 01 09:25:33.777264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.777230    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.777424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.777389    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpptf\" (UniqueName: \"kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.777472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.777443    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.777541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.777469    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.777605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.777588    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.777642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.777623    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879061    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpptf\" (UniqueName: \"kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879100    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879124    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879164    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879189    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879251    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879489    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879516    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.879742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.879682    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.881380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.881357    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.881773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.881754    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.935725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.935701    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpptf\" (UniqueName: \"kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf\") pod \"auth-test-model-46bf99-kserve-666d56fc8f-rg7th\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:33.936526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:33.936507    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:34.305006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:34.304983    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"]
Nov 01 09:25:34.306209 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:25:34.306183    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2eb30c5_5145_4f75_8084_ae7017186ce8.slice/crio-296c9923f25687a9be0d37198ecc96b3edd17d36536d6595abede96345824352 WatchSource:0}: Error finding container 296c9923f25687a9be0d37198ecc96b3edd17d36536d6595abede96345824352: Status 404 returned error can't find the container with id 296c9923f25687a9be0d37198ecc96b3edd17d36536d6595abede96345824352
Nov 01 09:25:34.816300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:34.816265    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th" event={"ID":"b2eb30c5-5145-4f75-8084-ae7017186ce8","Type":"ContainerStarted","Data":"f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de"}
Nov 01 09:25:34.816780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:34.816309    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th" event={"ID":"b2eb30c5-5145-4f75-8084-ae7017186ce8","Type":"ContainerStarted","Data":"296c9923f25687a9be0d37198ecc96b3edd17d36536d6595abede96345824352"}
Nov 01 09:25:34.876841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:34.876809    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:34.925675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:34.925622    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th" podStartSLOduration=1.9256092059999999 podStartE2EDuration="1.925609206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:25:34.925296258 +0000 UTC m=+3208.119690814" watchObservedRunningTime="2026-11-01 09:25:34.925609206 +0000 UTC m=+3208.120003759"
Nov 01 09:25:34.937483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:34.937450    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:39.642021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:39.641981    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:40.032365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:40.032335    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:40.044663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:40.044641    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:40.868751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:40.868722    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:25:41.322933 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:25:41.322899    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:25:43.936992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:43.936961    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:43.950017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:43.949990    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:44.412229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:44.412198    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:44.892360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:44.892329    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:25:49.205014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:49.204956    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:54.002706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:54.002676    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:58.818850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:58.818817    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:58.976056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:58.976019    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:25:59.176278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:25:59.176189    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:02.107675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.107642    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"]
Nov 01 09:26:02.112486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.112464    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.162508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.162481    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e40e82\"/\"openshift-service-ca.crt\""
Nov 01 09:26:02.190323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.190232    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.190323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.190268    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.190608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.190381    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.190608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.190424    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.190608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.190552    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.190608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.190591    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbf75\" (UniqueName: \"kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.208049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.208029    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e40e82\"/\"kube-root-ca.crt\""
Nov 01 09:26:02.208779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.208753    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e40e82\"/\"explicit-model-e40e82-kserve-self-signed-certs\""
Nov 01 09:26:02.208779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.208767    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e40e82\"/\"default-dockercfg-7fs5t\""
Nov 01 09:26:02.209273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.209250    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:02.211976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.211952    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"]
Nov 01 09:26:02.291552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291500    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.291747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291566    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.291747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291625    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.291747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291649    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbf75\" (UniqueName: \"kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.291903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291797    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.291903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291827    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.291996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.291947    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.292188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.292164    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.292254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.292220    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.293994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.293968    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.294126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.294107    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.348975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.348936    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbf75\" (UniqueName: \"kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75\") pod \"explicit-model-e40e82-kserve-544976f9f-7vw6d\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") " pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.425816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.425783    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:02.794597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:02.794567    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"]
Nov 01 09:26:02.795325 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:26:02.795298    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25922726_fe72_4d98_82bd_8b2ef763b9d7.slice/crio-85918ccced9ca6004a0db7f8a68287b6a0607eeb0094742d22d15982ec088676 WatchSource:0}: Error finding container 85918ccced9ca6004a0db7f8a68287b6a0607eeb0094742d22d15982ec088676: Status 404 returned error can't find the container with id 85918ccced9ca6004a0db7f8a68287b6a0607eeb0094742d22d15982ec088676
Nov 01 09:26:03.000107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:03.000071    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d" event={"ID":"25922726-fe72-4d98-82bd-8b2ef763b9d7","Type":"ContainerStarted","Data":"5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f"}
Nov 01 09:26:03.000277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:03.000115    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d" event={"ID":"25922726-fe72-4d98-82bd-8b2ef763b9d7","Type":"ContainerStarted","Data":"85918ccced9ca6004a0db7f8a68287b6a0607eeb0094742d22d15982ec088676"}
Nov 01 09:26:03.112430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:03.112328    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d" podStartSLOduration=1.112315197 podStartE2EDuration="1.112315197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:26:03.108442778 +0000 UTC m=+3236.302837332" watchObservedRunningTime="2026-11-01 09:26:03.112315197 +0000 UTC m=+3236.306709746"
Nov 01 09:26:03.426148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:03.426070    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:03.766365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:03.766335    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:08.533343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:08.533308    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:09.135262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:09.135230    4314 scope.go:122] "RemoveContainer" containerID="2f211f35b9abf94b0ab3b316dbe5b72be80e2f63a3ca7b33a964f17eac6cf345"
Nov 01 09:26:09.850572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:09.850515    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:11.322863 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:11.322829    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:26:12.426068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:12.426034    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:12.438645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:12.438623    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:13.073988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:13.073959    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:13.134752 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:13.134721    4314 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-self-signed-certs: secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:13.134941 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:13.134808    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs podName:25922726-fe72-4d98-82bd-8b2ef763b9d7 nodeName:}" failed. No retries permitted until 2026-11-01 09:26:13.634788285 +0000 UTC m=+3246.829182818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs") pod "explicit-model-e40e82-kserve-544976f9f-7vw6d" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7") : secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:13.241364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:13.241329    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"]
Nov 01 09:26:13.290032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:13.290002    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:13.480554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:13.480502    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:13.630206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:13.630159    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:13.642248 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:13.642223    4314 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-self-signed-certs: secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:13.642384 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:13.642288    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs podName:25922726-fe72-4d98-82bd-8b2ef763b9d7 nodeName:}" failed. No retries permitted until 2026-11-01 09:26:14.642275367 +0000 UTC m=+3247.836669900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs") pod "explicit-model-e40e82-kserve-544976f9f-7vw6d" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7") : secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:14.652619 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:14.652589    4314 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-self-signed-certs: secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:14.653003 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:14.652666    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs podName:25922726-fe72-4d98-82bd-8b2ef763b9d7 nodeName:}" failed. No retries permitted until 2026-11-01 09:26:16.652639623 +0000 UTC m=+3249.847034172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs") pod "explicit-model-e40e82-kserve-544976f9f-7vw6d" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7") : secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:16.678881 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:16.678844    4314 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-self-signed-certs: secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:16.679293 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:16.678924    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs podName:25922726-fe72-4d98-82bd-8b2ef763b9d7 nodeName:}" failed. No retries permitted until 2026-11-01 09:26:20.678908706 +0000 UTC m=+3253.873303257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs") pod "explicit-model-e40e82-kserve-544976f9f-7vw6d" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7") : secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:18.285661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:18.285604    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:18.493184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:18.493122    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:18.694502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:18.694415    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:20.733630 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:20.733595    4314 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-self-signed-certs: secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:20.734126 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:20.733688    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs podName:25922726-fe72-4d98-82bd-8b2ef763b9d7 nodeName:}" failed. No retries permitted until 2026-11-01 09:26:28.733668744 +0000 UTC m=+3261.928063277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs") pod "explicit-model-e40e82-kserve-544976f9f-7vw6d" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7") : secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:23.086312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:23.086279    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:27.874290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:27.874252    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:28.831370 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:28.831335    4314 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-self-signed-certs: secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:28.831644 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:28.831421    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs podName:25922726-fe72-4d98-82bd-8b2ef763b9d7 nodeName:}" failed. No retries permitted until 2026-11-01 09:26:44.83140199 +0000 UTC m=+3278.025796522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs") pod "explicit-model-e40e82-kserve-544976f9f-7vw6d" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7") : secret "explicit-model-e40e82-kserve-self-signed-certs" not found
Nov 01 09:26:30.097067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.097026    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d" podUID="25922726-fe72-4d98-82bd-8b2ef763b9d7" containerName="main" containerID="cri-o://5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f" gracePeriod=45
Nov 01 09:26:30.358628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.358570    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:30.450701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.450664    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm\") pod \"25922726-fe72-4d98-82bd-8b2ef763b9d7\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") "
Nov 01 09:26:30.450896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.450730    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs\") pod \"25922726-fe72-4d98-82bd-8b2ef763b9d7\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") "
Nov 01 09:26:30.450896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.450767    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75\" (UniqueName: \"kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75\") pod \"25922726-fe72-4d98-82bd-8b2ef763b9d7\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") "
Nov 01 09:26:30.450896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.450811    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home\") pod \"25922726-fe72-4d98-82bd-8b2ef763b9d7\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") "
Nov 01 09:26:30.450896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.450886    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache\") pod \"25922726-fe72-4d98-82bd-8b2ef763b9d7\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") "
Nov 01 09:26:30.451120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.450923    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir\") pod \"25922726-fe72-4d98-82bd-8b2ef763b9d7\" (UID: \"25922726-fe72-4d98-82bd-8b2ef763b9d7\") "
Nov 01 09:26:30.451172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.451141    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home" pod "25922726-fe72-4d98-82bd-8b2ef763b9d7" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:26:30.451476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.451453    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir" pod "25922726-fe72-4d98-82bd-8b2ef763b9d7" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:26:30.451609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.451486    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:26:30.451609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.451575    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache" pod "25922726-fe72-4d98-82bd-8b2ef763b9d7" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:26:30.453181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.453155    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs" pod "25922726-fe72-4d98-82bd-8b2ef763b9d7" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:26:30.453321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.453179    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm" pod "25922726-fe72-4d98-82bd-8b2ef763b9d7" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:26:30.453321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.453201    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75" pod "25922726-fe72-4d98-82bd-8b2ef763b9d7" (UID: "25922726-fe72-4d98-82bd-8b2ef763b9d7"). InnerVolumeSpecName "kube-api-access-rbf75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:26:30.552979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.552948    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:26:30.552979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.552974    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25922726-fe72-4d98-82bd-8b2ef763b9d7-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:26:30.552979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.552984    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbf75\" (UniqueName: \"kubernetes.io/projected/25922726-fe72-4d98-82bd-8b2ef763b9d7-kube-api-access-rbf75\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:26:30.553228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.552995    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:26:30.553228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.553004    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25922726-fe72-4d98-82bd-8b2ef763b9d7-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:26:30.738369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.738332    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"]
Nov 01 09:26:30.738967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.738951    4314 cpu_manager.go:414] "containerMap: removing container" podUID="25922726-fe72-4d98-82bd-8b2ef763b9d7" containerName="main"
Nov 01 09:26:30.739031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.738970    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25922726-fe72-4d98-82bd-8b2ef763b9d7" containerName="main"
Nov 01 09:26:30.739068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.739056    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="25922726-fe72-4d98-82bd-8b2ef763b9d7" containerName="main"
Nov 01 09:26:30.742669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.742654    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.792659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.792628    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a83714-openshift-default-dockercfg-6fhhb\""
Nov 01 09:26:30.846428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.846394    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"]
Nov 01 09:26:30.856155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856123    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856242    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856379    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856553    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856599    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856629    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856681    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhc85\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856791    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.856861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.856844    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.957884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.957847    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.957896    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.957968    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.957996    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958022    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958045    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc85\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958076    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958109    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958160    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958491    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958760    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958764    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958840    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.958988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.958918    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.960366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.960343    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:30.960708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:30.960689    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:31.024352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.024246    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhc85\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:31.026796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.026774    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:31.056621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.056594    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:31.184137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.184107    4314 generic.go:356] "Generic (PLEG): container finished" podID="25922726-fe72-4d98-82bd-8b2ef763b9d7" containerID="5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f" exitCode=0
Nov 01 09:26:31.184615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.184147    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"
Nov 01 09:26:31.184615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.184183    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d" event={"ID":"25922726-fe72-4d98-82bd-8b2ef763b9d7","Type":"ContainerDied","Data":"5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f"}
Nov 01 09:26:31.184615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.184231    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d" event={"ID":"25922726-fe72-4d98-82bd-8b2ef763b9d7","Type":"ContainerDied","Data":"85918ccced9ca6004a0db7f8a68287b6a0607eeb0094742d22d15982ec088676"}
Nov 01 09:26:31.184615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.184249    4314 scope.go:122] "RemoveContainer" containerID="5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f"
Nov 01 09:26:31.193787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.193767    4314 scope.go:122] "RemoveContainer" containerID="5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f"
Nov 01 09:26:31.194098 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:31.194077    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f\": container with ID starting with 5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f not found: ID does not exist" containerID="5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f"
Nov 01 09:26:31.194139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.194107    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f"} err="failed to get container status \"5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f\": rpc error: code = NotFound desc = could not find container \"5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f\": container with ID starting with 5651cdc4d373337f45865a31353a481132c2331678d3b1534a4d508711963c7f not found: ID does not exist"
Nov 01 09:26:31.339126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.339046    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"]
Nov 01 09:26:31.344835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.344809    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-e40e82/explicit-model-e40e82-kserve-544976f9f-7vw6d"]
Nov 01 09:26:31.496188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.496165    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"]
Nov 01 09:26:31.498026 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:26:31.497999    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce27064_0747_46ee_ba8f_3733ebfde94b.slice/crio-625da846cf491aa0bb05a1180da48949ce7db050bda2d96b3d68e62bf53ac393 WatchSource:0}: Error finding container 625da846cf491aa0bb05a1180da48949ce7db050bda2d96b3d68e62bf53ac393: Status 404 returned error can't find the container with id 625da846cf491aa0bb05a1180da48949ce7db050bda2d96b3d68e62bf53ac393
Nov 01 09:26:31.500993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.500957    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:26:31.501159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.501026    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:26:31.501159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:31.501059    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:26:32.193081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:32.193037    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" event={"ID":"3ce27064-0747-46ee-ba8f-3733ebfde94b","Type":"ContainerStarted","Data":"c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3"}
Nov 01 09:26:32.193589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:32.193101    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" event={"ID":"3ce27064-0747-46ee-ba8f-3733ebfde94b","Type":"ContainerStarted","Data":"625da846cf491aa0bb05a1180da48949ce7db050bda2d96b3d68e62bf53ac393"}
Nov 01 09:26:32.348678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:32.348621    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podStartSLOduration=2.348606849 podStartE2EDuration="2.348606849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:26:32.307421794 +0000 UTC m=+3265.501816348" watchObservedRunningTime="2026-11-01 09:26:32.348606849 +0000 UTC m=+3265.543001415"
Nov 01 09:26:32.720490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:32.720457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:32.929295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:32.929262    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25922726-fe72-4d98-82bd-8b2ef763b9d7" path="/var/lib/kubelet/pods/25922726-fe72-4d98-82bd-8b2ef763b9d7/volumes"
Nov 01 09:26:33.057460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:33.057430    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:33.082557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:33.082215    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 01 09:26:33.082557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:33.082339    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 01 09:26:34.058231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:34.057804    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 01 09:26:34.058231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:34.057865    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 01 09:26:35.057439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:35.057403    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 01 09:26:35.057681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:35.057452    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 01 09:26:36.056953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:36.056916    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 01 09:26:36.057350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:36.056970    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 01 09:26:37.057200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:37.057169    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 01 09:26:37.057706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:37.057675    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 01 09:26:37.620929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:37.620894    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:38.061872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:38.061842    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:38.062322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:38.062298    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:38.063021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:38.063001    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:26:41.326002 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:26:41.325968    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:26:42.427177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:42.427144    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:47.216942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:47.216909    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:48.907591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:48.907512    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:52.041520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:52.041488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:53.076374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:53.076344    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"]
Nov 01 09:26:53.121849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:53.121823    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a31d7-vm8w9"]
Nov 01 09:26:54.928785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:54.928752    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18def52f-4ef8-4e49-8d8b-da5ea74f3bd3" path="/var/lib/kubelet/pods/18def52f-4ef8-4e49-8d8b-da5ea74f3bd3/volumes"
Nov 01 09:26:55.086970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:55.086940    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"]
Nov 01 09:26:55.434757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:55.434721    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:55.591820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:55.591787    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:55.996518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:55.996484    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:26:57.107848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:26:57.107812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:00.087374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.087323    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj" podUID="017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" containerName="payload-processing" containerID="cri-o://7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d" gracePeriod=25
Nov 01 09:27:00.356640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.356611    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:27:00.367987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.367956    4314 generic.go:356] "Generic (PLEG): container finished" podID="017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" containerID="7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d" exitCode=0
Nov 01 09:27:00.368144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.367994    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"
Nov 01 09:27:00.368144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.368032    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj" event={"ID":"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9","Type":"ContainerDied","Data":"7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d"}
Nov 01 09:27:00.368144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.368109    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj" event={"ID":"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9","Type":"ContainerDied","Data":"bf32bd2068051b146c6216acc3245993b7c51e8ac7dce446923919fb9442b74f"}
Nov 01 09:27:00.368144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.368132    4314 scope.go:122] "RemoveContainer" containerID="7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d"
Nov 01 09:27:00.377681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.377663    4314 scope.go:122] "RemoveContainer" containerID="7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d"
Nov 01 09:27:00.377954 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:27:00.377929    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d\": container with ID starting with 7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d not found: ID does not exist" containerID="7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d"
Nov 01 09:27:00.378017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.377958    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d"} err="failed to get container status \"7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d\": rpc error: code = NotFound desc = could not find container \"7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d\": container with ID starting with 7b083c601ea71ae458170d9fe91f3594506da464f2051dd07761a7d32fc9a39d not found: ID does not exist"
Nov 01 09:27:00.424527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.424491    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume\") pod \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") "
Nov 01 09:27:00.424730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.424589    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28\" (UniqueName: \"kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28\") pod \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\" (UID: \"017f1b6e-123d-49ae-a8cc-c48bfbfa83d9\") "
Nov 01 09:27:00.424920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.424894    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume" pod "017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" (UID: "017f1b6e-123d-49ae-a8cc-c48bfbfa83d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:27:00.425065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.425040    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:00.426610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.426592    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28" pod "017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" (UID: "017f1b6e-123d-49ae-a8cc-c48bfbfa83d9"). InnerVolumeSpecName "kube-api-access-hqh28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:27:00.526027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.525992    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqh28\" (UniqueName: \"kubernetes.io/projected/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9-kube-api-access-hqh28\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:00.782932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.782895    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"]
Nov 01 09:27:00.786935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.786906    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e40e82-bf58f4445-2kdxj"]
Nov 01 09:27:00.930502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:00.930427    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" path="/var/lib/kubelet/pods/017f1b6e-123d-49ae-a8cc-c48bfbfa83d9/volumes"
Nov 01 09:27:01.402064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:01.402026    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:02.351251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:02.351204    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:03.203462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:03.203430    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:08.002375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:08.002328    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:08.175266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:08.175232    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:09.158695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:09.158664    4314 scope.go:122] "RemoveContainer" containerID="7f17bdd75c98e1cf1c84c4fb7c577abc12dacbe72b7a3ed6c74b1ae3d79930c3"
Nov 01 09:27:09.649226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:09.649192    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:10.265742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:10.265699    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"]
Nov 01 09:27:10.266242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:10.265993    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy" containerID="cri-o://7e7878e99d50e2bd231f3e822d50f73519a96c98b791935b38a97f281271e33b" gracePeriod=30
Nov 01 09:27:10.287598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:10.287567    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:10.440013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:10.439964    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:10.593773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:10.593680    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:10.891003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:10.890921    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:11.324924 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:27:11.324888    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:27:11.414028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.413989    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"]
Nov 01 09:27:11.414837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.414817    4314 cpu_manager.go:414] "containerMap: removing container" podUID="017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" containerName="payload-processing"
Nov 01 09:27:11.414893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.414840    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" containerName="payload-processing"
Nov 01 09:27:11.415020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.415008    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="017f1b6e-123d-49ae-a8cc-c48bfbfa83d9" containerName="payload-processing"
Nov 01 09:27:11.418916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.418894    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.498584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.498521    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"]
Nov 01 09:27:11.506273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.506239    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:11.516459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.516436    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-eb50a7-openshift-default-dockercfg-gq242\""
Nov 01 09:27:11.561042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561008    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561070    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561201    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561234    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561362    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8px\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561500    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561624    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561658    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.561841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.561736    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662514    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662623    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662661    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662683    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662734    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8px\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662811    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.662894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662863    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662897    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.662944    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.663265    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.663265    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.663330    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.663326    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.663471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.663434    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.665190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.665163    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.665411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.665390    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.720646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.720613    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8px\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.720646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.720636    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token\") pod \"e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") " pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.746645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.746608    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:11.750382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:11.750350    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:12.173159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.173129    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"]
Nov 01 09:27:12.206379 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:27:12.206352    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ad1e13_214f_4267_96de_e3d2eb28947a.slice/crio-7c7c2d0ea6a866bebb97725fde0d0ab8c565c935c75cc2c9c2d9f87635c0e095 WatchSource:0}: Error finding container 7c7c2d0ea6a866bebb97725fde0d0ab8c565c935c75cc2c9c2d9f87635c0e095: Status 404 returned error can't find the container with id 7c7c2d0ea6a866bebb97725fde0d0ab8c565c935c75cc2c9c2d9f87635c0e095
Nov 01 09:27:12.208930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.208896    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:27:12.209015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.208967    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:27:12.209015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.208996    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:27:12.462272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.462176    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k" event={"ID":"64ad1e13-214f-4267-96de-e3d2eb28947a","Type":"ContainerStarted","Data":"780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7"}
Nov 01 09:27:12.462272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.462245    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k" event={"ID":"64ad1e13-214f-4267-96de-e3d2eb28947a","Type":"ContainerStarted","Data":"7c7c2d0ea6a866bebb97725fde0d0ab8c565c935c75cc2c9c2d9f87635c0e095"}
Nov 01 09:27:12.747652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:12.747607    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:13.069557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:13.069457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:13.222664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:13.222630    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:13.388952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:13.388872    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:13.570165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:13.570134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:13.753186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:13.753156    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:13.944358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:13.944310    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k" podStartSLOduration=2.944295334 podStartE2EDuration="2.944295334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:27:12.577655877 +0000 UTC m=+3305.772050451" watchObservedRunningTime="2026-11-01 09:27:13.944295334 +0000 UTC m=+3307.138689888"
Nov 01 09:27:14.475284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:14.475256    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:14.476573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:14.476527    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:27:15.485975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.485934    4314 generic.go:356] "Generic (PLEG): container finished" podID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerID="7e7878e99d50e2bd231f3e822d50f73519a96c98b791935b38a97f281271e33b" exitCode=0
Nov 01 09:27:15.486483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.486010    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" event={"ID":"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f","Type":"ContainerDied","Data":"7e7878e99d50e2bd231f3e822d50f73519a96c98b791935b38a97f281271e33b"}
Nov 01 09:27:15.486483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.486093    4314 scope.go:122] "RemoveContainer" containerID="c627ed2e39bbd069c6dc5f7212c62ddb4c60a04a7ab419adcf82f24b7de0eedd"
Nov 01 09:27:15.529895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.529835    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:27:15.616643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.616612    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.616826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.616677    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.616826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.616713    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.616826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.616751    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.616955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.616934    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.616997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.616976    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.617049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617023    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:15.617100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617038    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:15.617100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617054    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.617191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617120    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:27:15.617234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617195    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.617234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617209    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:15.617350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617253    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token\") pod \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\" (UID: \"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f\") "
Nov 01 09:27:15.617495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617471    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:15.617934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.617913    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.618097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.618082    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.618204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.618184    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.618282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.618207    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.618282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.618221    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.619101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.619070    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:27:15.619597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.619575    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:27:15.619703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.619611    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "kube-api-access-2vcxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:27:15.619703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.619624    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy" pod "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" (UID: "9dbea5e9-84f4-4f79-a929-8e86d07d4d6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:15.719790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.719753    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.719790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.719785    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.720013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.719805    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:15.720013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:15.719816    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vcxd\" (UniqueName: \"kubernetes.io/projected/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f-kube-api-access-2vcxd\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:16.493644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:16.493609    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"
Nov 01 09:27:16.494107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:16.493640    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt" event={"ID":"9dbea5e9-84f4-4f79-a929-8e86d07d4d6f","Type":"ContainerDied","Data":"8af4eb0084ecd75ac567157fc56fa3fabafa5ad9a95bcadb512da8011a2c57dc"}
Nov 01 09:27:16.494107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:16.493704    4314 scope.go:122] "RemoveContainer" containerID="7e7878e99d50e2bd231f3e822d50f73519a96c98b791935b38a97f281271e33b"
Nov 01 09:27:16.694591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:16.694554    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"]
Nov 01 09:27:16.740714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:16.740678    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-e40e82-openshift-default-8b68b5f49-877zt"]
Nov 01 09:27:16.931266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:16.931186    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" path="/var/lib/kubelet/pods/9dbea5e9-84f4-4f79-a929-8e86d07d4d6f/volumes"
Nov 01 09:27:17.887251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:17.887217    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:18.686132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:18.686097    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:20.587098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:20.587064    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:22.979289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:22.979255    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"]
Nov 01 09:27:22.980025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:22.980001    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy"
Nov 01 09:27:22.980129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:22.980026    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy"
Nov 01 09:27:22.980200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:22.980186    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy"
Nov 01 09:27:22.984893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:22.984873    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.007346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.007314    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.007526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.007358    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.007526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.007518    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.007689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.007561    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5fm\" (UniqueName: \"kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.007689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.007614    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.007689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.007643    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.075897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.075864    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:23.079826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.079805    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-eb50a7\"/\"default-dockercfg-8d4p8\""
Nov 01 09:27:23.079932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.079829    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-eb50a7\"/\"openshift-service-ca.crt\""
Nov 01 09:27:23.079984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.079929    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-eb50a7\"/\"noten-model-eb50a7-kserve-self-signed-certs\""
Nov 01 09:27:23.079984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.079980    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-eb50a7\"/\"kube-root-ca.crt\""
Nov 01 09:27:23.083276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.083258    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"]
Nov 01 09:27:23.108853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.108823    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.108988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.108864    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5fm\" (UniqueName: \"kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.108988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.108925    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.108988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.108966    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.109167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.109131    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.109257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.109167    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.109345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.109322    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.109589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.109565    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.109589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.109596    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.112121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.112082    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.112321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.112303    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.170734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.170697    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5fm\" (UniqueName: \"kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm\") pod \"noten-model-eb50a7-kserve-765b456bc7-rdxcx\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") " pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.296992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.296958    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:23.532960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.532928    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:23.669755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:23.669724    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"]
Nov 01 09:27:23.670816 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:27:23.670790    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ebef252_4260_4bf7_992d_b81578a8e89f.slice/crio-97ff5ba84fb5bf026dc2c98575ef6af77c037b91e50bb74c794078c8905c8390 WatchSource:0}: Error finding container 97ff5ba84fb5bf026dc2c98575ef6af77c037b91e50bb74c794078c8905c8390: Status 404 returned error can't find the container with id 97ff5ba84fb5bf026dc2c98575ef6af77c037b91e50bb74c794078c8905c8390
Nov 01 09:27:24.547802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:24.547765    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx" event={"ID":"6ebef252-4260-4bf7-992d-b81578a8e89f","Type":"ContainerStarted","Data":"fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef"}
Nov 01 09:27:24.547802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:24.547811    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx" event={"ID":"6ebef252-4260-4bf7-992d-b81578a8e89f","Type":"ContainerStarted","Data":"97ff5ba84fb5bf026dc2c98575ef6af77c037b91e50bb74c794078c8905c8390"}
Nov 01 09:27:24.658392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:24.658334    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx" podStartSLOduration=2.658317989 podStartE2EDuration="2.658317989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:27:24.656616889 +0000 UTC m=+3317.851011444" watchObservedRunningTime="2026-11-01 09:27:24.658317989 +0000 UTC m=+3317.852712544"
Nov 01 09:27:25.297567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:25.297517    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:28.347164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:28.347135    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:33.297686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:33.297654    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:33.313398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:33.313368    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:33.325002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:33.324973    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:33.614842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:33.614758    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:27:38.042676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:38.042646    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:41.322808 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:27:41.322773    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:27:42.880433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:42.880396    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:47.796597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:47.796565    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:52.677479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:52.677437    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:27:57.589401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:27:57.589365    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:00.740900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:00.740866    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"]
Nov 01 09:28:00.741313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:00.741087    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k" podUID="64ad1e13-214f-4267-96de-e3d2eb28947a" containerName="istio-proxy" containerID="cri-o://780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7" gracePeriod=30
Nov 01 09:28:00.760933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:00.760900    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:01.112816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:01.112729    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:01.271282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:01.271249    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:02.386788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:02.386756    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:06.024759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.024693    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:28:06.122567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122525    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.122752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122582    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.122752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122633    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.122752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122669    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.122752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122708    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.123001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122762    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.123001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122818    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.123001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122930    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:06.123001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122942    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:06.123001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.122938    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.123287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123042    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data\") pod \"64ad1e13-214f-4267-96de-e3d2eb28947a\" (UID: \"64ad1e13-214f-4267-96de-e3d2eb28947a\") "
Nov 01 09:28:06.123287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123043    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:06.123287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123241    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:06.123466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123391    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:06.123790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123768    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ad1e13-214f-4267-96de-e3d2eb28947a-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.123976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123794    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.123976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123810    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.123976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123825    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.123976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.123838    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.125402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.125371    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:06.125526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.125438    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:06.125526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.125526    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "kube-api-access-fx8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:06.125858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.125841    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo" pod "64ad1e13-214f-4267-96de-e3d2eb28947a" (UID: "64ad1e13-214f-4267-96de-e3d2eb28947a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:28:06.211541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.211509    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"]
Nov 01 09:28:06.224824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.224793    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.224824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.224820    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.224990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.224835    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ad1e13-214f-4267-96de-e3d2eb28947a-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.224990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.224847    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8px\" (UniqueName: \"kubernetes.io/projected/64ad1e13-214f-4267-96de-e3d2eb28947a-kube-api-access-fx8px\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:06.298835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.298756    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:06.816075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.816043    4314 generic.go:356] "Generic (PLEG): container finished" podID="64ad1e13-214f-4267-96de-e3d2eb28947a" containerID="780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7" exitCode=0
Nov 01 09:28:06.816261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.816095    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"
Nov 01 09:28:06.816261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.816127    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k" event={"ID":"64ad1e13-214f-4267-96de-e3d2eb28947a","Type":"ContainerDied","Data":"780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7"}
Nov 01 09:28:06.816261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.816182    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k" event={"ID":"64ad1e13-214f-4267-96de-e3d2eb28947a","Type":"ContainerDied","Data":"7c7c2d0ea6a866bebb97725fde0d0ab8c565c935c75cc2c9c2d9f87635c0e095"}
Nov 01 09:28:06.816261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.816197    4314 scope.go:122] "RemoveContainer" containerID="780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7"
Nov 01 09:28:06.826505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.826489    4314 scope.go:122] "RemoveContainer" containerID="780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7"
Nov 01 09:28:06.826784 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:06.826764    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7\": container with ID starting with 780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7 not found: ID does not exist" containerID="780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7"
Nov 01 09:28:06.826840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.826792    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7"} err="failed to get container status \"780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7\": rpc error: code = NotFound desc = could not find container \"780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7\": container with ID starting with 780197e05f211a2a6e5f6b10f021b41458d46f0e7ebd7927b18172db7722b8f7 not found: ID does not exist"
Nov 01 09:28:06.972877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:06.972844    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"]
Nov 01 09:28:07.017319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:07.017287    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-eb50a7-openshift-default-55765cbd45-qtl8k"]
Nov 01 09:28:07.169674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:07.169588    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:08.928470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:08.928439    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ad1e13-214f-4267-96de-e3d2eb28947a" path="/var/lib/kubelet/pods/64ad1e13-214f-4267-96de-e3d2eb28947a/volumes"
Nov 01 09:28:09.190932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:09.190855    4314 scope.go:122] "RemoveContainer" containerID="41e55692e65aeaece951e9e8e926daef768129b05423fbb0a9875b8d1dbcc9df"
Nov 01 09:28:11.326643 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:11.325522    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:28:12.002368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.002333    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:12.281650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.281574    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"]
Nov 01 09:28:12.281820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.281782    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerName="discovery" containerID="cri-o://d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6" gracePeriod=30
Nov 01 09:28:12.560957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.560929    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:28:12.696428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696393    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.696632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696449    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.696632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696481    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.696632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696504    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.696632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696601    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.696913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696801    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.696913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696878    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert\") pod \"9afc937e-7f73-421c-8740-d7eafcfe12f1\" (UID: \"9afc937e-7f73-421c-8740-d7eafcfe12f1\") "
Nov 01 09:28:12.697038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.696922    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:12.697499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.697445    4314 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-ca-configmap\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.699199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.699160    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "kube-api-access-7qq4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:12.699199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.699187    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:28:12.699383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.699298    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:12.699700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.699674    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:28:12.699800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.699712    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:28:12.699800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.699721    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token" pod "9afc937e-7f73-421c-8740-d7eafcfe12f1" (UID: "9afc937e-7f73-421c-8740-d7eafcfe12f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:12.798901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.798824    4314 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-kubeconfig\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.798901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.798853    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.798901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.798865    4314 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afc937e-7f73-421c-8740-d7eafcfe12f1-local-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.798901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.798876    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qq4r\" (UniqueName: \"kubernetes.io/projected/9afc937e-7f73-421c-8740-d7eafcfe12f1-kube-api-access-7qq4r\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.798901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.798885    4314 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-cacerts\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.798901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.798894    4314 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9afc937e-7f73-421c-8740-d7eafcfe12f1-istio-csr-dns-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:12.857286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.857255    4314 generic.go:356] "Generic (PLEG): container finished" podID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerID="d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6" exitCode=0
Nov 01 09:28:12.857470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.857300    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"
Nov 01 09:28:12.857470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.857342    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" event={"ID":"9afc937e-7f73-421c-8740-d7eafcfe12f1","Type":"ContainerDied","Data":"d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6"}
Nov 01 09:28:12.857470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.857400    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" event={"ID":"9afc937e-7f73-421c-8740-d7eafcfe12f1","Type":"ContainerDied","Data":"c7c0de83101a04687100d0a5b6200d613b7a6f6196b94504ce3bd7d7c931de2a"}
Nov 01 09:28:12.857470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.857421    4314 scope.go:122] "RemoveContainer" containerID="d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6"
Nov 01 09:28:12.868197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.868029    4314 scope.go:122] "RemoveContainer" containerID="d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6"
Nov 01 09:28:12.868470 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:12.868452    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6\": container with ID starting with d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6 not found: ID does not exist" containerID="d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6"
Nov 01 09:28:12.868567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:12.868479    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6"} err="failed to get container status \"d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6\": rpc error: code = NotFound desc = could not find container \"d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6\": container with ID starting with d78d857f79d9c98c6c1db630c43a93903cf195a2942f770e61e27565d1094bf6 not found: ID does not exist"
Nov 01 09:28:13.011695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:13.011660    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"]
Nov 01 09:28:13.017879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:13.017855    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g"]
Nov 01 09:28:14.928937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:14.928905    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" path="/var/lib/kubelet/pods/9afc937e-7f73-421c-8740-d7eafcfe12f1/volumes"
Nov 01 09:28:16.776686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:16.776652    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:17.544559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:17.544511    4314 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dqd8g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:28:17.544733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:17.544603    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dqd8g" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.230:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:28:19.211318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:19.211235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:21.231665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.231607    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx" podUID="6ebef252-4260-4bf7-992d-b81578a8e89f" containerName="main" containerID="cri-o://fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef" gracePeriod=45
Nov 01 09:28:21.501149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.501126    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:28:21.582570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.582519    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:21.603313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603290    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home\") pod \"6ebef252-4260-4bf7-992d-b81578a8e89f\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") "
Nov 01 09:28:21.603444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603332    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm\") pod \"6ebef252-4260-4bf7-992d-b81578a8e89f\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") "
Nov 01 09:28:21.603492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603440    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir\") pod \"6ebef252-4260-4bf7-992d-b81578a8e89f\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") "
Nov 01 09:28:21.603492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603469    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache\") pod \"6ebef252-4260-4bf7-992d-b81578a8e89f\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") "
Nov 01 09:28:21.603616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603564    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs\") pod \"6ebef252-4260-4bf7-992d-b81578a8e89f\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") "
Nov 01 09:28:21.603616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603593    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home" pod "6ebef252-4260-4bf7-992d-b81578a8e89f" (UID: "6ebef252-4260-4bf7-992d-b81578a8e89f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:21.603736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603674    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm\" (UniqueName: \"kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm\") pod \"6ebef252-4260-4bf7-992d-b81578a8e89f\" (UID: \"6ebef252-4260-4bf7-992d-b81578a8e89f\") "
Nov 01 09:28:21.603736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603712    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir" pod "6ebef252-4260-4bf7-992d-b81578a8e89f" (UID: "6ebef252-4260-4bf7-992d-b81578a8e89f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:21.603836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.603727    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache" pod "6ebef252-4260-4bf7-992d-b81578a8e89f" (UID: "6ebef252-4260-4bf7-992d-b81578a8e89f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:21.604157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.604138    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:21.604242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.604158    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:21.604242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.604170    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:21.605651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.605631    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs" pod "6ebef252-4260-4bf7-992d-b81578a8e89f" (UID: "6ebef252-4260-4bf7-992d-b81578a8e89f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:28:21.605651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.605639    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm" pod "6ebef252-4260-4bf7-992d-b81578a8e89f" (UID: "6ebef252-4260-4bf7-992d-b81578a8e89f"). InnerVolumeSpecName "kube-api-access-8b5fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:21.605778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.605656    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm" pod "6ebef252-4260-4bf7-992d-b81578a8e89f" (UID: "6ebef252-4260-4bf7-992d-b81578a8e89f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:21.705183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.705151    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebef252-4260-4bf7-992d-b81578a8e89f-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:21.705183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.705179    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef252-4260-4bf7-992d-b81578a8e89f-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:21.705183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.705190    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b5fm\" (UniqueName: \"kubernetes.io/projected/6ebef252-4260-4bf7-992d-b81578a8e89f-kube-api-access-8b5fm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:21.915183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.915102    4314 generic.go:356] "Generic (PLEG): container finished" podID="6ebef252-4260-4bf7-992d-b81578a8e89f" containerID="fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef" exitCode=0
Nov 01 09:28:21.915183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.915152    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"
Nov 01 09:28:21.915379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.915187    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx" event={"ID":"6ebef252-4260-4bf7-992d-b81578a8e89f","Type":"ContainerDied","Data":"fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef"}
Nov 01 09:28:21.915379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.915233    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx" event={"ID":"6ebef252-4260-4bf7-992d-b81578a8e89f","Type":"ContainerDied","Data":"97ff5ba84fb5bf026dc2c98575ef6af77c037b91e50bb74c794078c8905c8390"}
Nov 01 09:28:21.915379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.915250    4314 scope.go:122] "RemoveContainer" containerID="fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef"
Nov 01 09:28:21.925710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.925692    4314 scope.go:122] "RemoveContainer" containerID="fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef"
Nov 01 09:28:21.925986 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:21.925968    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef\": container with ID starting with fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef not found: ID does not exist" containerID="fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef"
Nov 01 09:28:21.926030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:21.925995    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef"} err="failed to get container status \"fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef\": rpc error: code = NotFound desc = could not find container \"fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef\": container with ID starting with fdda563e3c7bd81f74760c30eec87af409e2372ef6b7da4b21e9c983e7a86cef not found: ID does not exist"
Nov 01 09:28:22.069431 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:22.069402    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"]
Nov 01 09:28:22.074152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:22.074131    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-eb50a7/noten-model-eb50a7-kserve-765b456bc7-rdxcx"]
Nov 01 09:28:22.929677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:22.929644    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebef252-4260-4bf7-992d-b81578a8e89f" path="/var/lib/kubelet/pods/6ebef252-4260-4bf7-992d-b81578a8e89f/volumes"
Nov 01 09:28:24.363854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.363822    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"]
Nov 01 09:28:24.364354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364338    4314 cpu_manager.go:414] "containerMap: removing container" podUID="64ad1e13-214f-4267-96de-e3d2eb28947a" containerName="istio-proxy"
Nov 01 09:28:24.364439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364355    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad1e13-214f-4267-96de-e3d2eb28947a" containerName="istio-proxy"
Nov 01 09:28:24.364439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364398    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6ebef252-4260-4bf7-992d-b81578a8e89f" containerName="main"
Nov 01 09:28:24.364439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364405    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebef252-4260-4bf7-992d-b81578a8e89f" containerName="main"
Nov 01 09:28:24.364439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364425    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerName="discovery"
Nov 01 09:28:24.364439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364431    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerName="discovery"
Nov 01 09:28:24.364613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364451    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy"
Nov 01 09:28:24.364613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364456    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy"
Nov 01 09:28:24.364613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364558    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afc937e-7f73-421c-8740-d7eafcfe12f1" containerName="discovery"
Nov 01 09:28:24.364613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364571    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad1e13-214f-4267-96de-e3d2eb28947a" containerName="istio-proxy"
Nov 01 09:28:24.364613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364579    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbea5e9-84f4-4f79-a929-8e86d07d4d6f" containerName="istio-proxy"
Nov 01 09:28:24.364613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.364586    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebef252-4260-4bf7-992d-b81578a8e89f" containerName="main"
Nov 01 09:28:24.369157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.369139    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.438347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.438310    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.438500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.438375    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.438500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.438456    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.438598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.438576    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.438641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.438635    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.438741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.438724    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w88n\" (UniqueName: \"kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.453118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.453093    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:24.464635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.464612    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-58de5a\"/\"openshift-service-ca.crt\""
Nov 01 09:28:24.464752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.464619    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-58de5a\"/\"kube-root-ca.crt\""
Nov 01 09:28:24.464752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.464648    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-58de5a\"/\"default-dockercfg-hmxlj\""
Nov 01 09:28:24.464855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.464698    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-58de5a\"/\"auth-test-model-58de5a-kserve-self-signed-certs\""
Nov 01 09:28:24.467981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.467960    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"]
Nov 01 09:28:24.540284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540253    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540306    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540342    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540434    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540521    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540614    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w88n\" (UniqueName: \"kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540736    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540809    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.540937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.540918    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.542650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.542629    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.542923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.542905    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.597735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.597705    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w88n\" (UniqueName: \"kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n\") pod \"auth-test-model-58de5a-kserve-957cf54fc-xwlw6\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") " pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.681994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.681920    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:24.736571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:24.736517    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:25.049950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:25.049922    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"]
Nov 01 09:28:25.050672 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:28:25.050646    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb67eb486_8b20_49ab_91e4_a621e4c4605b.slice/crio-7e9e83b1a8e3b0474af7a4f7ddb2c57542ec8f90a53d4058a35fc340cbe58ee0 WatchSource:0}: Error finding container 7e9e83b1a8e3b0474af7a4f7ddb2c57542ec8f90a53d4058a35fc340cbe58ee0: Status 404 returned error can't find the container with id 7e9e83b1a8e3b0474af7a4f7ddb2c57542ec8f90a53d4058a35fc340cbe58ee0
Nov 01 09:28:25.940720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:25.940685    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6" event={"ID":"b67eb486-8b20-49ab-91e4-a621e4c4605b","Type":"ContainerStarted","Data":"6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8"}
Nov 01 09:28:25.941140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:25.940731    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6" event={"ID":"b67eb486-8b20-49ab-91e4-a621e4c4605b","Type":"ContainerStarted","Data":"7e9e83b1a8e3b0474af7a4f7ddb2c57542ec8f90a53d4058a35fc340cbe58ee0"}
Nov 01 09:28:26.051756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:26.051691    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6" podStartSLOduration=2.051676836 podStartE2EDuration="2.051676836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:28:26.048947388 +0000 UTC m=+3379.243341943" watchObservedRunningTime="2026-11-01 09:28:26.051676836 +0000 UTC m=+3379.246071437"
Nov 01 09:28:26.412654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:26.412621    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:26.682689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:26.682609    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:31.272600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:31.272560    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:32.351625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:32.351586    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:34.682995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:34.682947    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:34.695045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:34.695016    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:35.003177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:35.003146    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:28:36.057477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:36.057444    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:40.871713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:40.871679    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:41.323745 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:41.323708    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:28:43.279543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.279497    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:43.515204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.515172    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7412-086b79"]
Nov 01 09:28:43.519290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.519267    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:43.553081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.553010    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcs69\" (UniqueName: \"kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69\") pod \"test-curl-7412-086b79\" (UID: \"6804e839-bf47-4871-8346-e699a361067e\") " pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:43.569170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.569143    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7412-086b79"]
Nov 01 09:28:43.611175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.611150    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-44wvn\""
Nov 01 09:28:43.653926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.653899    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs69\" (UniqueName: \"kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69\") pod \"test-curl-7412-086b79\" (UID: \"6804e839-bf47-4871-8346-e699a361067e\") " pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:43.710778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.710750    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcs69\" (UniqueName: \"kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69\") pod \"test-curl-7412-086b79\" (UID: \"6804e839-bf47-4871-8346-e699a361067e\") " pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:43.832623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:43.832548    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:44.200996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:44.200968    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7412-086b79"]
Nov 01 09:28:44.201920 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:28:44.201891    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6804e839_bf47_4871_8346_e699a361067e.slice/crio-a5d5bbb00479e4ad85bf07d4f36756ba3bc3925ca78052bfe6c70ae0b3ae31d3 WatchSource:0}: Error finding container a5d5bbb00479e4ad85bf07d4f36756ba3bc3925ca78052bfe6c70ae0b3ae31d3: Status 404 returned error can't find the container with id a5d5bbb00479e4ad85bf07d4f36756ba3bc3925ca78052bfe6c70ae0b3ae31d3
Nov 01 09:28:44.203822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:44.203803    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:28:45.052094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:45.052056    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7412-086b79" event={"ID":"6804e839-bf47-4871-8346-e699a361067e","Type":"ContainerStarted","Data":"a5d5bbb00479e4ad85bf07d4f36756ba3bc3925ca78052bfe6c70ae0b3ae31d3"}
Nov 01 09:28:45.669475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:45.669428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:48.072926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:48.072888    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7412-086b79" event={"ID":"6804e839-bf47-4871-8346-e699a361067e","Type":"ContainerStarted","Data":"b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d"}
Nov 01 09:28:48.179677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:48.179625    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7412-086b79" podStartSLOduration=2.311035881 podStartE2EDuration="5.179610576s" totalImagesPullingTime="2.868574695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:28:48.17851064 +0000 UTC m=+3401.372905196" watchObservedRunningTime="2026-11-01 09:28:48.179610576 +0000 UTC m=+3401.374005128"
Nov 01 09:28:49.647752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.647719    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7412-086b79"]
Nov 01 09:28:49.653824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.653795    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7412-086b79"]
Nov 01 09:28:49.810748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.810714    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7412-072e5c"]
Nov 01 09:28:49.811332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.811311    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6804e839-bf47-4871-8346-e699a361067e" containerName="test-curl-7412-086b79"
Nov 01 09:28:49.811437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.811333    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6804e839-bf47-4871-8346-e699a361067e" containerName="test-curl-7412-086b79"
Nov 01 09:28:49.811507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.811491    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6804e839-bf47-4871-8346-e699a361067e" containerName="test-curl-7412-086b79"
Nov 01 09:28:49.814979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.814960    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:49.865959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.865936    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7412-072e5c"]
Nov 01 09:28:49.925016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:49.924954    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vfd5\" (UniqueName: \"kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5\") pod \"test-curl-7412-072e5c\" (UID: \"da2da385-db33-47d6-94a8-7a8edd40d97b\") " pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:50.026597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:50.026563    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vfd5\" (UniqueName: \"kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5\") pod \"test-curl-7412-072e5c\" (UID: \"da2da385-db33-47d6-94a8-7a8edd40d97b\") " pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:50.085122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:50.085087    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vfd5\" (UniqueName: \"kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5\") pod \"test-curl-7412-072e5c\" (UID: \"da2da385-db33-47d6-94a8-7a8edd40d97b\") " pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:50.086090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:50.086065    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7412-086b79" podUID="6804e839-bf47-4871-8346-e699a361067e" containerName="test-curl-7412-086b79" containerID="cri-o://b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d" gracePeriod=2
Nov 01 09:28:50.126722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:50.126694    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:50.133599 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:50.133570    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7412-086b79\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="6804e839-bf47-4871-8346-e699a361067e" pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:50.507111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:50.507070    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:50.515003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:50.514965    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7412-072e5c"]
Nov 01 09:28:50.516608 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:28:50.516567    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda2da385_db33_47d6_94a8_7a8edd40d97b.slice/crio-e193b3a85ec55337c9661f8ff425064e7bb7ca7dfc4359949edc0cfb150213a8 WatchSource:0}: Error finding container e193b3a85ec55337c9661f8ff425064e7bb7ca7dfc4359949edc0cfb150213a8: Status 404 returned error can't find the container with id e193b3a85ec55337c9661f8ff425064e7bb7ca7dfc4359949edc0cfb150213a8
Nov 01 09:28:51.093546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:51.093506    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7412-072e5c" event={"ID":"da2da385-db33-47d6-94a8-7a8edd40d97b","Type":"ContainerStarted","Data":"3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6"}
Nov 01 09:28:51.093954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:51.093579    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7412-072e5c" event={"ID":"da2da385-db33-47d6-94a8-7a8edd40d97b","Type":"ContainerStarted","Data":"e193b3a85ec55337c9661f8ff425064e7bb7ca7dfc4359949edc0cfb150213a8"}
Nov 01 09:28:51.202193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:51.202138    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7412-072e5c" podStartSLOduration=2.202123558 podStartE2EDuration="2.202123558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:28:51.198452782 +0000 UTC m=+3404.392847337" watchObservedRunningTime="2026-11-01 09:28:51.202123558 +0000 UTC m=+3404.396518161"
Nov 01 09:28:52.337991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.337964    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:52.460700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.460620    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69\" (UniqueName: \"kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69\") pod \"6804e839-bf47-4871-8346-e699a361067e\" (UID: \"6804e839-bf47-4871-8346-e699a361067e\") "
Nov 01 09:28:52.462857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.462832    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69" pod "6804e839-bf47-4871-8346-e699a361067e" (UID: "6804e839-bf47-4871-8346-e699a361067e"). InnerVolumeSpecName "kube-api-access-lcs69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:52.562905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.562869    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcs69\" (UniqueName: \"kubernetes.io/projected/6804e839-bf47-4871-8346-e699a361067e-kube-api-access-lcs69\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:52.728098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.728020    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7412-072e5c"]
Nov 01 09:28:52.732522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.732494    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7412-072e5c"]
Nov 01 09:28:52.929679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:52.929647    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6804e839-bf47-4871-8346-e699a361067e" path="/var/lib/kubelet/pods/6804e839-bf47-4871-8346-e699a361067e/volumes"
Nov 01 09:28:53.106937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.106907    4314 generic.go:356] "Generic (PLEG): container finished" podID="6804e839-bf47-4871-8346-e699a361067e" containerID="b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d" exitCode=137
Nov 01 09:28:53.107150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.106954    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-086b79"
Nov 01 09:28:53.107150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.107018    4314 scope.go:122] "RemoveContainer" containerID="b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d"
Nov 01 09:28:53.107276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.107183    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7412-072e5c" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" containerName="test-curl-7412-072e5c" containerID="cri-o://3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6" gracePeriod=2
Nov 01 09:28:53.116043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.116023    4314 scope.go:122] "RemoveContainer" containerID="b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d"
Nov 01 09:28:53.116303 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:53.116275    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d\": container with ID starting with b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d not found: ID does not exist" containerID="b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d"
Nov 01 09:28:53.116366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.116309    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d"} err="failed to get container status \"b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d\": rpc error: code = NotFound desc = could not find container \"b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d\": container with ID starting with b8d16dee1bce370951e186674c0c0d1c024cd76f9405f7d19fd1a0b66d55a44d not found: ID does not exist"
Nov 01 09:28:53.527579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:53.527516    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:54.035783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.035751    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"]
Nov 01 09:28:54.036383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.036364    4314 cpu_manager.go:414] "containerMap: removing container" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" containerName="test-curl-7412-072e5c"
Nov 01 09:28:54.036451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.036384    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" containerName="test-curl-7412-072e5c"
Nov 01 09:28:54.036489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.036470    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" containerName="test-curl-7412-072e5c"
Nov 01 09:28:54.041368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.041347    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.100267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.100229    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-54c25e-openshift-default-dockercfg-8xb5k\""
Nov 01 09:28:54.129004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.128971    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:54.150646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.150610    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"]
Nov 01 09:28:54.182224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182184    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182254    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182292    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5q62\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182342    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182398    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182495    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182550    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182598    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.182760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.182702    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284018    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284060    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284100    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284185    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284237    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284278    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284321    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q62\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284367    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284418    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284478    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284495    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284873    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.284953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.284921    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.285030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.285005    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.286622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.286565    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.286787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.286770    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.341416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.341381    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.341610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.341466    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q62\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62\") pod \"e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") " pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.354456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.354431    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:54.359349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.359325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:54.633472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.633398    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"]
Nov 01 09:28:54.638460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.638442    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.687609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.687583    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:28:54.687793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.687764    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:28:54.688158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.688136    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.688377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.688334    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.688377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.688370    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64jz\" (UniqueName: \"kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.691519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.691497    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"]
Nov 01 09:28:54.697564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.696446    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:28:54.697564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.696657    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:28:54.772234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.772211    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"]
Nov 01 09:28:54.773224 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:28:54.773201    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d797966_8bdd_49fc_8c2e_f20b3f5ffe06.slice/crio-a4193ac24ef08cd4b04f53f8bb16d4273b23fb7010006d30b407178785c15abf WatchSource:0}: Error finding container a4193ac24ef08cd4b04f53f8bb16d4273b23fb7010006d30b407178785c15abf: Status 404 returned error can't find the container with id a4193ac24ef08cd4b04f53f8bb16d4273b23fb7010006d30b407178785c15abf
Nov 01 09:28:54.775793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.775763    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:28:54.775865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.775819    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:28:54.775865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.775858    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:28:54.790040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.790020    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.790186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.790170    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.790267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.790197    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64jz\" (UniqueName: \"kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.790863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.790843    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.792960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.792939    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.846973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.846941    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64jz\" (UniqueName: \"kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz\") pod \"maas-api-revoke-sub-54354543-c74mm\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:54.951896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:54.951812    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:55.140109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.140064    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" event={"ID":"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06","Type":"ContainerStarted","Data":"c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"}
Nov 01 09:28:55.140248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.140141    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" event={"ID":"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06","Type":"ContainerStarted","Data":"a4193ac24ef08cd4b04f53f8bb16d4273b23fb7010006d30b407178785c15abf"}
Nov 01 09:28:55.254139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.254074    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podStartSLOduration=2.254056781 podStartE2EDuration="2.254056781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:28:55.252082567 +0000 UTC m=+3408.446477123" watchObservedRunningTime="2026-11-01 09:28:55.254056781 +0000 UTC m=+3408.448451337"
Nov 01 09:28:55.326974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.326944    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"]
Nov 01 09:28:55.332119 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:28:55.332090    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf353989a_652c_41ad_8d3f_1b1f517742e3.slice/crio-376cc397efd7dfefec7432bdf8669ed0ec13d815a6fb259295526457de24d31a WatchSource:0}: Error finding container 376cc397efd7dfefec7432bdf8669ed0ec13d815a6fb259295526457de24d31a: Status 404 returned error can't find the container with id 376cc397efd7dfefec7432bdf8669ed0ec13d815a6fb259295526457de24d31a
Nov 01 09:28:55.350984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.350953    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:55.354895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.354875    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:55.361425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.361407    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:55.384098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.384075    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:55.396668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.396643    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5\" (UniqueName: \"kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5\") pod \"da2da385-db33-47d6-94a8-7a8edd40d97b\" (UID: \"da2da385-db33-47d6-94a8-7a8edd40d97b\") "
Nov 01 09:28:55.398592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.398573    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5" pod "da2da385-db33-47d6-94a8-7a8edd40d97b" (UID: "da2da385-db33-47d6-94a8-7a8edd40d97b"). InnerVolumeSpecName "kube-api-access-2vfd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:55.498554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:55.498502    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vfd5\" (UniqueName: \"kubernetes.io/projected/da2da385-db33-47d6-94a8-7a8edd40d97b-kube-api-access-2vfd5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:56.147516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.147480    4314 generic.go:356] "Generic (PLEG): container finished" podID="da2da385-db33-47d6-94a8-7a8edd40d97b" containerID="3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6" exitCode=137
Nov 01 09:28:56.148001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.147525    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:56.148001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.147585    4314 scope.go:122] "RemoveContainer" containerID="3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6"
Nov 01 09:28:56.149359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.149317    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm" event={"ID":"f353989a-652c-41ad-8d3f-1b1f517742e3","Type":"ContainerStarted","Data":"0012c1f280338f25dd634a8116de9d55130afe736d9e34625a04015bbaf14de1"}
Nov 01 09:28:56.149480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.149388    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm" event={"ID":"f353989a-652c-41ad-8d3f-1b1f517742e3","Type":"ContainerStarted","Data":"376cc397efd7dfefec7432bdf8669ed0ec13d815a6fb259295526457de24d31a"}
Nov 01 09:28:56.149864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.149835    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:56.150711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.150691    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:28:56.157729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.157711    4314 scope.go:122] "RemoveContainer" containerID="3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6"
Nov 01 09:28:56.157991 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:56.157972    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6\": container with ID starting with 3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6 not found: ID does not exist" containerID="3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6"
Nov 01 09:28:56.158062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.157999    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6"} err="failed to get container status \"3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6\": rpc error: code = NotFound desc = could not find container \"3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6\": container with ID starting with 3f205aa76bb3af8704dcb953e6d066fd64f712ec4d933d10de36d43bd37495f6 not found: ID does not exist"
Nov 01 09:28:56.259109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.259065    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm" podStartSLOduration=2.259050506 podStartE2EDuration="2.259050506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:28:56.256330041 +0000 UTC m=+3409.450724600" watchObservedRunningTime="2026-11-01 09:28:56.259050506 +0000 UTC m=+3409.453445060"
Nov 01 09:28:56.303582 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:56.303552    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7412-072e5c\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:56.929056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:56.929022    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" path="/var/lib/kubelet/pods/da2da385-db33-47d6-94a8-7a8edd40d97b/volumes"
Nov 01 09:28:56.972391 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:28:56.972355    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7412-072e5c\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="da2da385-db33-47d6-94a8-7a8edd40d97b" pod="opendatahub/test-curl-7412-072e5c"
Nov 01 09:28:57.158907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:57.158870    4314 generic.go:356] "Generic (PLEG): container finished" podID="f353989a-652c-41ad-8d3f-1b1f517742e3" containerID="0012c1f280338f25dd634a8116de9d55130afe736d9e34625a04015bbaf14de1" exitCode=0
Nov 01 09:28:57.159275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:57.158921    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm" event={"ID":"f353989a-652c-41ad-8d3f-1b1f517742e3","Type":"ContainerDied","Data":"0012c1f280338f25dd634a8116de9d55130afe736d9e34625a04015bbaf14de1"}
Nov 01 09:28:58.215932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.215896    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:58.323107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.323080    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:28:58.379926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.379894    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:58.433214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.433149    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca\") pod \"f353989a-652c-41ad-8d3f-1b1f517742e3\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") "
Nov 01 09:28:58.433445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.433420    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz\" (UniqueName: \"kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz\") pod \"f353989a-652c-41ad-8d3f-1b1f517742e3\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") "
Nov 01 09:28:58.433705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.433684    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca\") pod \"f353989a-652c-41ad-8d3f-1b1f517742e3\" (UID: \"f353989a-652c-41ad-8d3f-1b1f517742e3\") "
Nov 01 09:28:58.434580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.434552    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca" pod "f353989a-652c-41ad-8d3f-1b1f517742e3" (UID: "f353989a-652c-41ad-8d3f-1b1f517742e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:58.436265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.436245    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz" pod "f353989a-652c-41ad-8d3f-1b1f517742e3" (UID: "f353989a-652c-41ad-8d3f-1b1f517742e3"). InnerVolumeSpecName "kube-api-access-t64jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:58.436894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.436868    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca" pod "f353989a-652c-41ad-8d3f-1b1f517742e3" (UID: "f353989a-652c-41ad-8d3f-1b1f517742e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:28:58.523250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.523216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:28:58.535692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.535665    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64jz\" (UniqueName: \"kubernetes.io/projected/f353989a-652c-41ad-8d3f-1b1f517742e3-kube-api-access-t64jz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:58.535759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.535692    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:58.535759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:58.535702    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f353989a-652c-41ad-8d3f-1b1f517742e3-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:59.172747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:59.172706    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm" event={"ID":"f353989a-652c-41ad-8d3f-1b1f517742e3","Type":"ContainerDied","Data":"376cc397efd7dfefec7432bdf8669ed0ec13d815a6fb259295526457de24d31a"}
Nov 01 09:28:59.172951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:59.172764    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="376cc397efd7dfefec7432bdf8669ed0ec13d815a6fb259295526457de24d31a"
Nov 01 09:28:59.172951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:28:59.172719    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"
Nov 01 09:29:00.415363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:00.415317    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:01.358570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.358491    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:01.526010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.525971    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:01.759173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.759140    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"]
Nov 01 09:29:01.759702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.759687    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f353989a-652c-41ad-8d3f-1b1f517742e3" containerName="revoke-keys"
Nov 01 09:29:01.759757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.759703    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f353989a-652c-41ad-8d3f-1b1f517742e3" containerName="revoke-keys"
Nov 01 09:29:01.759824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.759815    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f353989a-652c-41ad-8d3f-1b1f517742e3" containerName="revoke-keys"
Nov 01 09:29:01.763359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.763340    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:01.816123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.816097    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"]
Nov 01 09:29:01.863462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.863431    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-54c25e-dockercfg-5fj59\""
Nov 01 09:29:01.865561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.863809    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-54c25e\""
Nov 01 09:29:01.882652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.882618    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:01.883109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.883065    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzk5\" (UniqueName: \"kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5\") pod \"payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:01.923047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.923015    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"]
Nov 01 09:29:01.927008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.926993    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:01.980885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.980859    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"]
Nov 01 09:29:01.984996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.984967    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzk5\" (UniqueName: \"kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5\") pod \"payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:01.985165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.985146    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:01.985835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:01.985814    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:02.050362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.050296    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzk5\" (UniqueName: \"kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5\") pod \"payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:02.076920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.076893    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:02.087060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.087032    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbf7\" (UniqueName: \"kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7\") pod \"payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.087168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.087100    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.188167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.188129    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbf7\" (UniqueName: \"kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7\") pod \"payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.188369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.188199    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.189167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.189138    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.245545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.245491    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbf7\" (UniqueName: \"kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7\") pod \"payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.453030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.452997    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"]
Nov 01 09:29:02.453731 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:02.453704    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb99506_a507_4313_86c6_9b47e9ed920d.slice/crio-90f904ff08ead6d64e58616ed8bcdbde26f5dc9eda148dd677a9932ec181c47b WatchSource:0}: Error finding container 90f904ff08ead6d64e58616ed8bcdbde26f5dc9eda148dd677a9932ec181c47b: Status 404 returned error can't find the container with id 90f904ff08ead6d64e58616ed8bcdbde26f5dc9eda148dd677a9932ec181c47b
Nov 01 09:29:02.540231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.540207    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:02.909256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:02.909225    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"]
Nov 01 09:29:02.909881 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:02.909853    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bd9eb6_4d64_4b7c_9533_e8a2ac7ebc7d.slice/crio-f76ca11bd0db4879898baf4d49e13c19f63002657aef6edb0d3e3a210802b7ec WatchSource:0}: Error finding container f76ca11bd0db4879898baf4d49e13c19f63002657aef6edb0d3e3a210802b7ec: Status 404 returned error can't find the container with id f76ca11bd0db4879898baf4d49e13c19f63002657aef6edb0d3e3a210802b7ec
Nov 01 09:29:03.199517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.199419    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7" event={"ID":"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d","Type":"ContainerStarted","Data":"45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6"}
Nov 01 09:29:03.199517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.199482    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7" event={"ID":"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d","Type":"ContainerStarted","Data":"f76ca11bd0db4879898baf4d49e13c19f63002657aef6edb0d3e3a210802b7ec"}
Nov 01 09:29:03.199517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.199510    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:03.201152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.201117    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4" event={"ID":"6cb99506-a507-4313-86c6-9b47e9ed920d","Type":"ContainerStarted","Data":"7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916"}
Nov 01 09:29:03.201278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.201157    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4" event={"ID":"6cb99506-a507-4313-86c6-9b47e9ed920d","Type":"ContainerStarted","Data":"90f904ff08ead6d64e58616ed8bcdbde26f5dc9eda148dd677a9932ec181c47b"}
Nov 01 09:29:03.201278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.201199    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:03.309802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.309750    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7" podStartSLOduration=2.309736522 podStartE2EDuration="2.309736522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:29:03.308496878 +0000 UTC m=+3416.502891433" watchObservedRunningTime="2026-11-01 09:29:03.309736522 +0000 UTC m=+3416.504131077"
Nov 01 09:29:03.418752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:03.418694    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4" podStartSLOduration=2.418680327 podStartE2EDuration="2.418680327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:29:03.416020318 +0000 UTC m=+3416.610414869" watchObservedRunningTime="2026-11-01 09:29:03.418680327 +0000 UTC m=+3416.613074882"
Nov 01 09:29:05.414169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.414134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:05.416208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.416181    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"]
Nov 01 09:29:05.420319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.420301    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.515899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.515863    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:29:05.516118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.515969    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:29:05.516118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.515981    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:29:05.516118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.515968    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:29:05.519401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.519381    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"]
Nov 01 09:29:05.528122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.528096    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.528250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.528231    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxr6\" (UniqueName: \"kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.528368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.528353    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.630157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.630122    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxr6\" (UniqueName: \"kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.630355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.630225    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.630355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.630300    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.631057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.631029    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.632721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.632700    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.687174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.687117    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxr6\" (UniqueName: \"kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6\") pod \"maas-api-revoke-sub-1ecb47ab-4dtxt\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:05.734317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:05.734288    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:06.100743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:06.100713    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"]
Nov 01 09:29:06.101672 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:06.101644    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46de1d0e_0f0c_437e_9e55_03acf5373160.slice/crio-ec0a762f3516d8ecd30cf11ab8db4f72ae9e3fc54eae2c8b0e17fe58bf36b516 WatchSource:0}: Error finding container ec0a762f3516d8ecd30cf11ab8db4f72ae9e3fc54eae2c8b0e17fe58bf36b516: Status 404 returned error can't find the container with id ec0a762f3516d8ecd30cf11ab8db4f72ae9e3fc54eae2c8b0e17fe58bf36b516
Nov 01 09:29:06.222936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:06.222900    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt" event={"ID":"46de1d0e-0f0c-437e-9e55-03acf5373160","Type":"ContainerStarted","Data":"ec0a762f3516d8ecd30cf11ab8db4f72ae9e3fc54eae2c8b0e17fe58bf36b516"}
Nov 01 09:29:07.230749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:07.230715    4314 generic.go:356] "Generic (PLEG): container finished" podID="46de1d0e-0f0c-437e-9e55-03acf5373160" containerID="af67317600103b96dcd70ac4e5d4938c55a6bcaf8c433be8cedaaf0e830fc25c" exitCode=0
Nov 01 09:29:07.231160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:07.230803    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt" event={"ID":"46de1d0e-0f0c-437e-9e55-03acf5373160","Type":"ContainerDied","Data":"af67317600103b96dcd70ac4e5d4938c55a6bcaf8c433be8cedaaf0e830fc25c"}
Nov 01 09:29:08.389077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.389052    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:08.463781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.463755    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca\") pod \"46de1d0e-0f0c-437e-9e55-03acf5373160\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") "
Nov 01 09:29:08.463932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.463810    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca\") pod \"46de1d0e-0f0c-437e-9e55-03acf5373160\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") "
Nov 01 09:29:08.464065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.464048    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6\" (UniqueName: \"kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6\") pod \"46de1d0e-0f0c-437e-9e55-03acf5373160\" (UID: \"46de1d0e-0f0c-437e-9e55-03acf5373160\") "
Nov 01 09:29:08.464152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.464131    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca" pod "46de1d0e-0f0c-437e-9e55-03acf5373160" (UID: "46de1d0e-0f0c-437e-9e55-03acf5373160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:29:08.464464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.464445    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:08.465934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.465904    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca" pod "46de1d0e-0f0c-437e-9e55-03acf5373160" (UID: "46de1d0e-0f0c-437e-9e55-03acf5373160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:29:08.466040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.465958    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6" pod "46de1d0e-0f0c-437e-9e55-03acf5373160" (UID: "46de1d0e-0f0c-437e-9e55-03acf5373160"). InnerVolumeSpecName "kube-api-access-hkxr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:29:08.565720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.565647    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkxr6\" (UniqueName: \"kubernetes.io/projected/46de1d0e-0f0c-437e-9e55-03acf5373160-kube-api-access-hkxr6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:08.565720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:08.565675    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46de1d0e-0f0c-437e-9e55-03acf5373160-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:09.244669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:09.244643    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"
Nov 01 09:29:09.244787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:09.244676    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt" event={"ID":"46de1d0e-0f0c-437e-9e55-03acf5373160","Type":"ContainerDied","Data":"ec0a762f3516d8ecd30cf11ab8db4f72ae9e3fc54eae2c8b0e17fe58bf36b516"}
Nov 01 09:29:09.244787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:09.244734    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec0a762f3516d8ecd30cf11ab8db4f72ae9e3fc54eae2c8b0e17fe58bf36b516"
Nov 01 09:29:10.285602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:10.285564    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:11.327680 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:29:11.327643    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:29:14.210858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:14.210830    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:29:14.211362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:14.210896    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:29:15.082178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:15.082148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:19.527184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:19.527149    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:19.886513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:19.886417    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:24.714454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:24.714405    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:29.470232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:29.470178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:34.235009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:34.234964    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:36.358917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:36.358878    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:36.502760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:36.502726    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:39.102687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:39.102656    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:41.325778 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:29:41.325733    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:29:43.965793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:43.965763    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:45.240869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.240830    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"]
Nov 01 09:29:45.241692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.241669    4314 cpu_manager.go:414] "containerMap: removing container" podUID="46de1d0e-0f0c-437e-9e55-03acf5373160" containerName="revoke-keys"
Nov 01 09:29:45.241692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.241691    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46de1d0e-0f0c-437e-9e55-03acf5373160" containerName="revoke-keys"
Nov 01 09:29:45.241898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.241801    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="46de1d0e-0f0c-437e-9e55-03acf5373160" containerName="revoke-keys"
Nov 01 09:29:45.245154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.245136    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.316339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.316302    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.316555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.316359    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.316555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.316454    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.316555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.316488    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.316776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.316567    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbn7m\" (UniqueName: \"kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.316776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.316605    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.325898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.325869    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"]
Nov 01 09:29:45.337263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.337228    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:45.340444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.340422    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a83714\"/\"openshift-service-ca.crt\""
Nov 01 09:29:45.340608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.340501    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a83714\"/\"kube-root-ca.crt\""
Nov 01 09:29:45.340608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.340508    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a83714\"/\"default-dockercfg-zwj7s\""
Nov 01 09:29:45.340720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.340587    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a83714\"/\"auth-test-model-a83714-kserve-self-signed-certs\""
Nov 01 09:29:45.417660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.417624    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.417844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.417683    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.417844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.417730    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.417929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.417846    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.417975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.417950    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbn7m\" (UniqueName: \"kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.418021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.417994    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.418171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.418151    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.418258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.418179    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.418258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.418234    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.420258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.420231    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.420735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.420708    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.477146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.477111    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbn7m\" (UniqueName: \"kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m\") pod \"auth-test-model-a83714-kserve-559f7c59bf-bc4c8\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") " pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.558456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.558374    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:45.821882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.821800    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"]
Nov 01 09:29:45.826125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.826099    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:45.875216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.875190    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:29:45.875216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.875193    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:29:45.879731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.879709    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"]
Nov 01 09:29:45.920887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.920867    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:29:45.921024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.920939    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:29:45.923651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.923595    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:45.923651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.923647    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csndb\" (UniqueName: \"kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:45.923877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.923699    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:45.927611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:45.927592    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"]
Nov 01 09:29:45.956600 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:45.956571    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8499201a_74b6_4f23_b97f_66a43f5df7d4.slice/crio-3da2ae3d4bf2ba57e40737859b909e4fbc7ec18bce8f208ca1010b32b3dc257d WatchSource:0}: Error finding container 3da2ae3d4bf2ba57e40737859b909e4fbc7ec18bce8f208ca1010b32b3dc257d: Status 404 returned error can't find the container with id 3da2ae3d4bf2ba57e40737859b909e4fbc7ec18bce8f208ca1010b32b3dc257d
Nov 01 09:29:46.024991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.024965    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.025091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.025009    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csndb\" (UniqueName: \"kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.025200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.025169    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.025922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.025900    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.027309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.027285    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.081871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.081811    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csndb\" (UniqueName: \"kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb\") pod \"maas-api-revoke-sub-40c94d39-c6ccb\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.143380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.143349    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:46.488576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.488514    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8" event={"ID":"8499201a-74b6-4f23-b97f-66a43f5df7d4","Type":"ContainerStarted","Data":"66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881"}
Nov 01 09:29:46.488576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.488577    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8" event={"ID":"8499201a-74b6-4f23-b97f-66a43f5df7d4","Type":"ContainerStarted","Data":"3da2ae3d4bf2ba57e40737859b909e4fbc7ec18bce8f208ca1010b32b3dc257d"}
Nov 01 09:29:46.515671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.515643    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"]
Nov 01 09:29:46.516713 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:46.516683    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99627d1_4d31_4b0c_a031_943afa967ad1.slice/crio-65bebf9ca85e4441240712fc55f098c9da34cf6329b7e198c6d11ba2e279410b WatchSource:0}: Error finding container 65bebf9ca85e4441240712fc55f098c9da34cf6329b7e198c6d11ba2e279410b: Status 404 returned error can't find the container with id 65bebf9ca85e4441240712fc55f098c9da34cf6329b7e198c6d11ba2e279410b
Nov 01 09:29:46.558789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.558763    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:46.599711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:46.599663    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8" podStartSLOduration=1.599649516 podStartE2EDuration="1.599649516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:29:46.597469783 +0000 UTC m=+3459.791864338" watchObservedRunningTime="2026-11-01 09:29:46.599649516 +0000 UTC m=+3459.794044071"
Nov 01 09:29:47.498111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.498085    4314 generic.go:356] "Generic (PLEG): container finished" podID="a99627d1-4d31-4b0c-a031-943afa967ad1" containerID="1645fa07501a44b0c7952c6cb69df47c7f27fc12a4ca55662f867a465e17ad66" exitCode=0
Nov 01 09:29:47.498473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.498164    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb" event={"ID":"a99627d1-4d31-4b0c-a031-943afa967ad1","Type":"ContainerDied","Data":"1645fa07501a44b0c7952c6cb69df47c7f27fc12a4ca55662f867a465e17ad66"}
Nov 01 09:29:47.498473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.498214    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb" event={"ID":"a99627d1-4d31-4b0c-a031-943afa967ad1","Type":"ContainerStarted","Data":"65bebf9ca85e4441240712fc55f098c9da34cf6329b7e198c6d11ba2e279410b"}
Nov 01 09:29:47.584171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.584133    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"]
Nov 01 09:29:47.588679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.588662    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.629803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.629769    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"]
Nov 01 09:29:47.747124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.747092    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwbf\" (UniqueName: \"kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.747311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.747239    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.747365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.747331    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.848415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.848383    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwbf\" (UniqueName: \"kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.848600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.848492    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.848600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.848574    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.849178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.849155    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.851004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.850979    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:47.905938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:47.905915    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwbf\" (UniqueName: \"kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf\") pod \"maas-api-revoke-sub-8db45573-hf25j\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:48.202998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.202924    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:48.570061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.570034    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"]
Nov 01 09:29:48.578450 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:48.578419    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b292ca_98a5_45f8_9c89_8f6be656b3c6.slice/crio-9de3c07dd2f027b2d9d51cad9c0a45bdabdd34c92ac86ee1a4ddd6579bfb8c83 WatchSource:0}: Error finding container 9de3c07dd2f027b2d9d51cad9c0a45bdabdd34c92ac86ee1a4ddd6579bfb8c83: Status 404 returned error can't find the container with id 9de3c07dd2f027b2d9d51cad9c0a45bdabdd34c92ac86ee1a4ddd6579bfb8c83
Nov 01 09:29:48.662836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.662813    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:48.759975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.759917    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb\" (UniqueName: \"kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb\") pod \"a99627d1-4d31-4b0c-a031-943afa967ad1\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") "
Nov 01 09:29:48.760108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.760032    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca\") pod \"a99627d1-4d31-4b0c-a031-943afa967ad1\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") "
Nov 01 09:29:48.760204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.760185    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca\") pod \"a99627d1-4d31-4b0c-a031-943afa967ad1\" (UID: \"a99627d1-4d31-4b0c-a031-943afa967ad1\") "
Nov 01 09:29:48.760458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.760430    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca" pod "a99627d1-4d31-4b0c-a031-943afa967ad1" (UID: "a99627d1-4d31-4b0c-a031-943afa967ad1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:29:48.760732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.760710    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:48.762429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.762408    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca" pod "a99627d1-4d31-4b0c-a031-943afa967ad1" (UID: "a99627d1-4d31-4b0c-a031-943afa967ad1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:29:48.762512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.762462    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb" pod "a99627d1-4d31-4b0c-a031-943afa967ad1" (UID: "a99627d1-4d31-4b0c-a031-943afa967ad1"). InnerVolumeSpecName "kube-api-access-csndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:29:48.809394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.809369    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:48.862495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.862464    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99627d1-4d31-4b0c-a031-943afa967ad1-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:48.862495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:48.862494    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csndb\" (UniqueName: \"kubernetes.io/projected/a99627d1-4d31-4b0c-a031-943afa967ad1-kube-api-access-csndb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:49.513294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:49.513261    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"
Nov 01 09:29:49.513497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:49.513287    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb" event={"ID":"a99627d1-4d31-4b0c-a031-943afa967ad1","Type":"ContainerDied","Data":"65bebf9ca85e4441240712fc55f098c9da34cf6329b7e198c6d11ba2e279410b"}
Nov 01 09:29:49.513497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:49.513341    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65bebf9ca85e4441240712fc55f098c9da34cf6329b7e198c6d11ba2e279410b"
Nov 01 09:29:49.515082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:49.515056    4314 generic.go:356] "Generic (PLEG): container finished" podID="c0b292ca-98a5-45f8-9c89-8f6be656b3c6" containerID="c0cd4af34c6249d946a3d1dcdfc82b65dd159cdb76e7245e46b2a73f7bc87120" exitCode=0
Nov 01 09:29:49.515203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:49.515141    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j" event={"ID":"c0b292ca-98a5-45f8-9c89-8f6be656b3c6","Type":"ContainerDied","Data":"c0cd4af34c6249d946a3d1dcdfc82b65dd159cdb76e7245e46b2a73f7bc87120"}
Nov 01 09:29:49.515203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:49.515182    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j" event={"ID":"c0b292ca-98a5-45f8-9c89-8f6be656b3c6","Type":"ContainerStarted","Data":"9de3c07dd2f027b2d9d51cad9c0a45bdabdd34c92ac86ee1a4ddd6579bfb8c83"}
Nov 01 09:29:50.672570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.672526    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:50.785181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.785111    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf\" (UniqueName: \"kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf\") pod \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") "
Nov 01 09:29:50.785337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.785274    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca\") pod \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") "
Nov 01 09:29:50.785337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.785308    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca\") pod \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\" (UID: \"c0b292ca-98a5-45f8-9c89-8f6be656b3c6\") "
Nov 01 09:29:50.785804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.785774    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca" pod "c0b292ca-98a5-45f8-9c89-8f6be656b3c6" (UID: "c0b292ca-98a5-45f8-9c89-8f6be656b3c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:29:50.787264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.787236    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf" pod "c0b292ca-98a5-45f8-9c89-8f6be656b3c6" (UID: "c0b292ca-98a5-45f8-9c89-8f6be656b3c6"). InnerVolumeSpecName "kube-api-access-wfwbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:29:50.787368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.787336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca" pod "c0b292ca-98a5-45f8-9c89-8f6be656b3c6" (UID: "c0b292ca-98a5-45f8-9c89-8f6be656b3c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:29:50.887141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.887107    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:50.887141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.887140    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:50.887141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:50.887156    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfwbf\" (UniqueName: \"kubernetes.io/projected/c0b292ca-98a5-45f8-9c89-8f6be656b3c6-kube-api-access-wfwbf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:51.530070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:51.530040    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"
Nov 01 09:29:51.530275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:51.530082    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j" event={"ID":"c0b292ca-98a5-45f8-9c89-8f6be656b3c6","Type":"ContainerDied","Data":"9de3c07dd2f027b2d9d51cad9c0a45bdabdd34c92ac86ee1a4ddd6579bfb8c83"}
Nov 01 09:29:51.530275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:51.530180    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de3c07dd2f027b2d9d51cad9c0a45bdabdd34c92ac86ee1a4ddd6579bfb8c83"
Nov 01 09:29:53.577106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:53.577071    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:55.558550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:55.558497    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:55.570924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:55.570903    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:56.570653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.570622    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:29:56.580236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.580211    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"]
Nov 01 09:29:56.580863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.580842    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a99627d1-4d31-4b0c-a031-943afa967ad1" containerName="revoke-keys"
Nov 01 09:29:56.580976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.580864    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99627d1-4d31-4b0c-a031-943afa967ad1" containerName="revoke-keys"
Nov 01 09:29:56.580976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.580914    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c0b292ca-98a5-45f8-9c89-8f6be656b3c6" containerName="revoke-keys"
Nov 01 09:29:56.580976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.580923    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b292ca-98a5-45f8-9c89-8f6be656b3c6" containerName="revoke-keys"
Nov 01 09:29:56.581143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.581079    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99627d1-4d31-4b0c-a031-943afa967ad1" containerName="revoke-keys"
Nov 01 09:29:56.581143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.581100    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b292ca-98a5-45f8-9c89-8f6be656b3c6" containerName="revoke-keys"
Nov 01 09:29:56.584857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.584838    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.634079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.633825    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:29:56.634079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.633995    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:29:56.653678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.653645    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.653842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.653684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.654020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.654001    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl47p\" (UniqueName: \"kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.664942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.664920    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:29:56.665050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.665010    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:29:56.670743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.670723    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"]
Nov 01 09:29:56.755359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.755327    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.755567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.755368    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.755649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.755613    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl47p\" (UniqueName: \"kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.756098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.756072    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.757616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.757595    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.812963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.812926    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl47p\" (UniqueName: \"kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p\") pod \"maas-api-revoke-sub-2cc2b9cc-2zhs7\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:56.898526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:56.898447    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:57.005803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:57.005776    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:57.274580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:57.274550    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"]
Nov 01 09:29:57.274856 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:57.274835    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297f0dff_623f_43e2_bf73_39cd1c121a91.slice/crio-c1cbf6d44216cefab320fa271ae9a96359379cc48d37d36536a31d9d33a352c9 WatchSource:0}: Error finding container c1cbf6d44216cefab320fa271ae9a96359379cc48d37d36536a31d9d33a352c9: Status 404 returned error can't find the container with id c1cbf6d44216cefab320fa271ae9a96359379cc48d37d36536a31d9d33a352c9
Nov 01 09:29:57.570135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:57.570107    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7" event={"ID":"297f0dff-623f-43e2-bf73-39cd1c121a91","Type":"ContainerStarted","Data":"c1cbf6d44216cefab320fa271ae9a96359379cc48d37d36536a31d9d33a352c9"}
Nov 01 09:29:58.364736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.364661    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"]
Nov 01 09:29:58.368801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.368783    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.385029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.385001    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk6p\" (UniqueName: \"kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.385144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.385072    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.385196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.385148    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.469469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.469440    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"]
Nov 01 09:29:58.486844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.486811    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk6p\" (UniqueName: \"kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.486962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.486906    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.487027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.486974    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.487802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.487775    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.490057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.490032    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.538822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.538604    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:29:58.548172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.548144    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk6p\" (UniqueName: \"kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p\") pod \"maas-api-revoke-sub-06a3be1e-m95v2\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:58.578900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.578871    4314 generic.go:356] "Generic (PLEG): container finished" podID="297f0dff-623f-43e2-bf73-39cd1c121a91" containerID="56a6fb343dbcf3d808e0f9bd47f45edb1f10771dd18a6a51a9c44a62ebae76cf" exitCode=0
Nov 01 09:29:58.579066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.578956    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7" event={"ID":"297f0dff-623f-43e2-bf73-39cd1c121a91","Type":"ContainerDied","Data":"56a6fb343dbcf3d808e0f9bd47f45edb1f10771dd18a6a51a9c44a62ebae76cf"}
Nov 01 09:29:58.681852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:58.681781    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:29:59.052184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.052155    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"]
Nov 01 09:29:59.053041 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:29:59.053013    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af3f87b_9e47_4054_a99d_d4e7e113bab8.slice/crio-11b159bf3c1b2869221fe5ecdaefe061bcfca2068832f0ec88f5752bcd7a9c25 WatchSource:0}: Error finding container 11b159bf3c1b2869221fe5ecdaefe061bcfca2068832f0ec88f5752bcd7a9c25: Status 404 returned error can't find the container with id 11b159bf3c1b2869221fe5ecdaefe061bcfca2068832f0ec88f5752bcd7a9c25
Nov 01 09:29:59.359339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.359247    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"]
Nov 01 09:29:59.364772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.364748    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.398972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.398945    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.398980    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399162    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399222    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399306    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spq6z\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399360    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399379    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.399480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.399461    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.417976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.417950    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-65aeef-openshift-default-dockercfg-5fxbb\""
Nov 01 09:29:59.464651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.464624    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"]
Nov 01 09:29:59.500323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500282    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500370    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spq6z\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500418    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500442    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500489    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500594    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500624    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500672    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500744    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.500969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.500939    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.501048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.501011    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.501293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.501264    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.501668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.501643    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.503110 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.503090    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.503373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.503352    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.557144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.557122    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.557268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.557250    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spq6z\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.587599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.587573    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2" event={"ID":"5af3f87b-9e47-4054-a99d-d4e7e113bab8","Type":"ContainerStarted","Data":"9b78fc1c1e1e5412a1668d03a983bac1e0e315d5340432206cebf88f17cecdef"}
Nov 01 09:29:59.587711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.587627    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2" event={"ID":"5af3f87b-9e47-4054-a99d-d4e7e113bab8","Type":"ContainerStarted","Data":"11b159bf3c1b2869221fe5ecdaefe061bcfca2068832f0ec88f5752bcd7a9c25"}
Nov 01 09:29:59.678427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.678397    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:29:59.695690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.695636    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2" podStartSLOduration=1.695616986 podStartE2EDuration="1.695616986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:29:59.694741521 +0000 UTC m=+3472.889136077" watchObservedRunningTime="2026-11-01 09:29:59.695616986 +0000 UTC m=+3472.890011541"
Nov 01 09:29:59.719243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.719217    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:29:59.804178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.804149    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p\" (UniqueName: \"kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p\") pod \"297f0dff-623f-43e2-bf73-39cd1c121a91\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") "
Nov 01 09:29:59.804301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.804246    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca\") pod \"297f0dff-623f-43e2-bf73-39cd1c121a91\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") "
Nov 01 09:29:59.804360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.804310    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca\") pod \"297f0dff-623f-43e2-bf73-39cd1c121a91\" (UID: \"297f0dff-623f-43e2-bf73-39cd1c121a91\") "
Nov 01 09:29:59.804657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.804631    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca" pod "297f0dff-623f-43e2-bf73-39cd1c121a91" (UID: "297f0dff-623f-43e2-bf73-39cd1c121a91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:29:59.804836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.804815    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:59.806237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.806215    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p" pod "297f0dff-623f-43e2-bf73-39cd1c121a91" (UID: "297f0dff-623f-43e2-bf73-39cd1c121a91"). InnerVolumeSpecName "kube-api-access-nl47p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:29:59.806326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.806254    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca" pod "297f0dff-623f-43e2-bf73-39cd1c121a91" (UID: "297f0dff-623f-43e2-bf73-39cd1c121a91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:29:59.906103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.906030    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297f0dff-623f-43e2-bf73-39cd1c121a91-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:29:59.906103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:29:59.906058    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl47p\" (UniqueName: \"kubernetes.io/projected/297f0dff-623f-43e2-bf73-39cd1c121a91-kube-api-access-nl47p\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:00.099707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.099673    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"]
Nov 01 09:30:00.099959 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:00.099938    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf4b813_2a29_4707_b5fa_e72b028fa369.slice/crio-fa2a40b88436ccee638c00dcc3ec9d10b6623d6df68d9e9033414c401ef0519a WatchSource:0}: Error finding container fa2a40b88436ccee638c00dcc3ec9d10b6623d6df68d9e9033414c401ef0519a: Status 404 returned error can't find the container with id fa2a40b88436ccee638c00dcc3ec9d10b6623d6df68d9e9033414c401ef0519a
Nov 01 09:30:00.102621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.102588    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:00.102683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.102659    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:00.102719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.102687    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:00.159807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.159736    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"]
Nov 01 09:30:00.160584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.160563    4314 cpu_manager.go:414] "containerMap: removing container" podUID="297f0dff-623f-43e2-bf73-39cd1c121a91" containerName="revoke-keys"
Nov 01 09:30:00.160584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.160583    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297f0dff-623f-43e2-bf73-39cd1c121a91" containerName="revoke-keys"
Nov 01 09:30:00.160735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.160722    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="297f0dff-623f-43e2-bf73-39cd1c121a91" containerName="revoke-keys"
Nov 01 09:30:00.164081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.164064    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"]
Nov 01 09:30:00.164178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.164164    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.170574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.168688    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"]
Nov 01 09:30:00.170574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.168799    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.173254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.173234    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"]
Nov 01 09:30:00.173358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.173338    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.177464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.177447    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"]
Nov 01 09:30:00.177599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.177584    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.181402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.181380    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.188787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.188764    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"]
Nov 01 09:30:00.202196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.202156    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.210823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.207583    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"]
Nov 01 09:30:00.211803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.211597    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bs5\" (UniqueName: \"kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.211803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.211691    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.211976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.211924    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwq2\" (UniqueName: \"kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.212498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.212387    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.212498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.212459    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkdh\" (UniqueName: \"kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.212677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.212549    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bzb\" (UniqueName: \"kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.212677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.212592    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.212995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.212684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9qj\" (UniqueName: \"kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.216572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.214178    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.216572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.214564    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.216572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.214617    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.216572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.215355    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"]
Nov 01 09:30:00.221696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.221677    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"]
Nov 01 09:30:00.221806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.221785    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.230719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.230699    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"]
Nov 01 09:30:00.230817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.230739    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"]
Nov 01 09:30:00.230859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.230818    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.282653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.276217    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"]
Nov 01 09:30:00.287743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.287596    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"]
Nov 01 09:30:00.288600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.288521    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"]
Nov 01 09:30:00.294627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.294600    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"]
Nov 01 09:30:00.296111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.296034    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"]
Nov 01 09:30:00.302580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.301999    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"]
Nov 01 09:30:00.315860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.315840    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwq2\" (UniqueName: \"kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.316012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316000    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rv9r\" (UniqueName: \"kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.316116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316106    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.316209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316197    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.316276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316267    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshr8\" (UniqueName: \"kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.316382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316372    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkdh\" (UniqueName: \"kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.316453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316444    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bzb\" (UniqueName: \"kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.316511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316503    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.316645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316591    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.316714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316645    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.316768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316712    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmn2\" (UniqueName: \"kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.316768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316738    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9qj\" (UniqueName: \"kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.316768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316762    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.316932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316789    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.316932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316822    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bs5\" (UniqueName: \"kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.316932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316851    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.316932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316879    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.316932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.316900    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xwj\" (UniqueName: \"kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.317692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.317670    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.317911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.317891    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.318277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.318257    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.318395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.318378    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.318611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.318589    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.342390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.342367    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"]
Nov 01 09:30:00.374939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.374901    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bs5\" (UniqueName: \"kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.375350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.375083    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkdh\" (UniqueName: \"kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.375350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.375173    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bzb\" (UniqueName: \"kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.375868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.375843    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwq2\" (UniqueName: \"kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.376477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.376461    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9qj\" (UniqueName: \"kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.417819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.417763    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rv9r\" (UniqueName: \"kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.417819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.417809    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.418042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.417831    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshr8\" (UniqueName: \"kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.418042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.417880    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.418042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.417897    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.418042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.417948    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmn2\" (UniqueName: \"kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.418042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.418040    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.418392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.418089    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xwj\" (UniqueName: \"kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.418579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.418559    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.418665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.418645    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.418709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.418689    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.418780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.418761    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.475341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.475314    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rv9r\" (UniqueName: \"kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.475598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.475580    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xwj\" (UniqueName: \"kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.475722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.475702    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshr8\" (UniqueName: \"kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.475895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.475880    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmn2\" (UniqueName: \"kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.480407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.480390    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:00.489918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.489897    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:00.514183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.514153    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:00.520749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.519298    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:00.529282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.529254    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:00.538733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.538710    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:00.552440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.548606    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:00.576595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.576429    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:00.607576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.605319    4314 generic.go:356] "Generic (PLEG): container finished" podID="5af3f87b-9e47-4054-a99d-d4e7e113bab8" containerID="9b78fc1c1e1e5412a1668d03a983bac1e0e315d5340432206cebf88f17cecdef" exitCode=0
Nov 01 09:30:00.607576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.605494    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2" event={"ID":"5af3f87b-9e47-4054-a99d-d4e7e113bab8","Type":"ContainerDied","Data":"9b78fc1c1e1e5412a1668d03a983bac1e0e315d5340432206cebf88f17cecdef"}
Nov 01 09:30:00.608161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.608131    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7" event={"ID":"297f0dff-623f-43e2-bf73-39cd1c121a91","Type":"ContainerDied","Data":"c1cbf6d44216cefab320fa271ae9a96359379cc48d37d36536a31d9d33a352c9"}
Nov 01 09:30:00.608268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.608188    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cbf6d44216cefab320fa271ae9a96359379cc48d37d36536a31d9d33a352c9"
Nov 01 09:30:00.608347 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.608284    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"
Nov 01 09:30:00.612383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.612344    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:00.616970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.616157    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" event={"ID":"caf4b813-2a29-4707-b5fa-e72b028fa369","Type":"ContainerStarted","Data":"98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01"}
Nov 01 09:30:00.616970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.616213    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" event={"ID":"caf4b813-2a29-4707-b5fa-e72b028fa369","Type":"ContainerStarted","Data":"fa2a40b88436ccee638c00dcc3ec9d10b6623d6df68d9e9033414c401ef0519a"}
Nov 01 09:30:00.678870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.678799    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:30:00.976473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:00.976438    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"]
Nov 01 09:30:01.023735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.023705    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"]
Nov 01 09:30:01.024638 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:01.024614    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b74d268_7fef_47fc_a995_fec1203c7e11.slice/crio-ede86aa921fcef620db3860f9930587f1b37b506e3bf1d926f0c45921c59a577 WatchSource:0}: Error finding container ede86aa921fcef620db3860f9930587f1b37b506e3bf1d926f0c45921c59a577: Status 404 returned error can't find the container with id ede86aa921fcef620db3860f9930587f1b37b506e3bf1d926f0c45921c59a577
Nov 01 09:30:01.030954 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:01.030933    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad05b42_95fc_4704_861d_ce903c61cff9.slice/crio-56c4b5e67c48a3e339ad86d5058bf563524d1a9461995873976700895d70f616 WatchSource:0}: Error finding container 56c4b5e67c48a3e339ad86d5058bf563524d1a9461995873976700895d70f616: Status 404 returned error can't find the container with id 56c4b5e67c48a3e339ad86d5058bf563524d1a9461995873976700895d70f616
Nov 01 09:30:01.033082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.032987    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"]
Nov 01 09:30:01.089868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.089843    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"]
Nov 01 09:30:01.093264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.093242    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"]
Nov 01 09:30:01.100130 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:01.100103    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8849cd98_018b_4ff4_a9a0_02cf3ce8d86a.slice/crio-3f50004ab7e5cc9af0ffd0886fffceda687e5da6acd8082447d3060a9cbdf539 WatchSource:0}: Error finding container 3f50004ab7e5cc9af0ffd0886fffceda687e5da6acd8082447d3060a9cbdf539: Status 404 returned error can't find the container with id 3f50004ab7e5cc9af0ffd0886fffceda687e5da6acd8082447d3060a9cbdf539
Nov 01 09:30:01.100977 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:01.100952    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31989828_2bfe_49ac_a171_540ec24af1c0.slice/crio-d585a2e68fd9dbec4afbc15d47a2b22d1ef524f9bcbaca7c219c783cc07a31ea WatchSource:0}: Error finding container d585a2e68fd9dbec4afbc15d47a2b22d1ef524f9bcbaca7c219c783cc07a31ea: Status 404 returned error can't find the container with id d585a2e68fd9dbec4afbc15d47a2b22d1ef524f9bcbaca7c219c783cc07a31ea
Nov 01 09:30:01.120954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.120884    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"]
Nov 01 09:30:01.126330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.126285    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"]
Nov 01 09:30:01.127032 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:01.127002    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76c0b14_cede_4b3a_80e0_cdc8beaf9724.slice/crio-b4c746029537efa848f23a9dc6c0be71e6ce4005b9b8137e6d9e19440431516e WatchSource:0}: Error finding container b4c746029537efa848f23a9dc6c0be71e6ce4005b9b8137e6d9e19440431516e: Status 404 returned error can't find the container with id b4c746029537efa848f23a9dc6c0be71e6ce4005b9b8137e6d9e19440431516e
Nov 01 09:30:01.217776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.217752    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"]
Nov 01 09:30:01.292728 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:01.292689    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode434ec4a_df75_4ae0_89bc_a20ff6c3d726.slice/crio-ad0bd9d12dad9ae24869b929a12510dda4aa40224d18d9655e4696c954ff3cfd WatchSource:0}: Error finding container ad0bd9d12dad9ae24869b929a12510dda4aa40224d18d9655e4696c954ff3cfd: Status 404 returned error can't find the container with id ad0bd9d12dad9ae24869b929a12510dda4aa40224d18d9655e4696c954ff3cfd
Nov 01 09:30:01.293295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.292991    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podStartSLOduration=2.292974488 podStartE2EDuration="2.292974488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:01.279917818 +0000 UTC m=+3474.474312374" watchObservedRunningTime="2026-11-01 09:30:01.292974488 +0000 UTC m=+3474.487369044"
Nov 01 09:30:01.297002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.296977    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"]
Nov 01 09:30:01.624409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.624381    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" event={"ID":"5294f372-7697-4725-9642-3e189f9244a4","Type":"ContainerStarted","Data":"1e2abe6708a0ec5cfc74bf27079f22f8b68cf59c098a947a5e2d052a6adcd2dc"}
Nov 01 09:30:01.625690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.625666    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerStarted","Data":"f5fb4de9da34c1f9870ee962e5ef0e3a162f2e46d3ec1d55069fef5c66f6b673"}
Nov 01 09:30:01.626767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.626748    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerStarted","Data":"ad0bd9d12dad9ae24869b929a12510dda4aa40224d18d9655e4696c954ff3cfd"}
Nov 01 09:30:01.627869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.627850    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" event={"ID":"7b74d268-7fef-47fc-a995-fec1203c7e11","Type":"ContainerStarted","Data":"ede86aa921fcef620db3860f9930587f1b37b506e3bf1d926f0c45921c59a577"}
Nov 01 09:30:01.629517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.629484    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerStarted","Data":"18be8f5ba4bb77a4b63a417c6d5df48ccd70e2e6807fec2a35ecb74d13bf5201"}
Nov 01 09:30:01.629620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.629527    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerStarted","Data":"d585a2e68fd9dbec4afbc15d47a2b22d1ef524f9bcbaca7c219c783cc07a31ea"}
Nov 01 09:30:01.631130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.631110    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerStarted","Data":"69fc9cbdaf60e18ed89ea783e31bb5e5447cad647c8294063c986514d989c11f"}
Nov 01 09:30:01.631209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.631145    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerStarted","Data":"b71f973321413488c809b596c26db83ac300d1b9791bfaf8e94076aecfaef256"}
Nov 01 09:30:01.632783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.632758    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerStarted","Data":"ef142fa3350a26da40f6fe8786b426a47b2fac37d28040a2076a44a3cf0117f9"}
Nov 01 09:30:01.632893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.632815    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerStarted","Data":"b4c746029537efa848f23a9dc6c0be71e6ce4005b9b8137e6d9e19440431516e"}
Nov 01 09:30:01.634208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.634181    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerStarted","Data":"df7854cd33483f440563c0df0b331ddc0a911e70d5d63e1ae6192e03d9715183"}
Nov 01 09:30:01.634298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.634221    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerStarted","Data":"3f50004ab7e5cc9af0ffd0886fffceda687e5da6acd8082447d3060a9cbdf539"}
Nov 01 09:30:01.635702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.635680    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerStarted","Data":"c8b0b4cb49f0dff6adb1ad62e7a8dbb43a952a3982d5383b40864ec93dcf28a2"}
Nov 01 09:30:01.635798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.635728    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerStarted","Data":"56c4b5e67c48a3e339ad86d5058bf563524d1a9461995873976700895d70f616"}
Nov 01 09:30:01.690920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.690873    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 01 09:30:01.691189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.690963    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 01 09:30:01.742463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.742378    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" podStartSLOduration=1.742363941 podStartE2EDuration="1.742363941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:01.739260936 +0000 UTC m=+3474.933655510" watchObservedRunningTime="2026-11-01 09:30:01.742363941 +0000 UTC m=+3474.936758496"
Nov 01 09:30:01.854812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.854765    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" podStartSLOduration=1.8547512369999999 podStartE2EDuration="1.854751237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:01.851139907 +0000 UTC m=+3475.045534478" watchObservedRunningTime="2026-11-01 09:30:01.854751237 +0000 UTC m=+3475.049145792"
Nov 01 09:30:01.895088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.895065    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:30:01.942949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.942922    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca\") pod \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") "
Nov 01 09:30:01.943055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.943035    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p\" (UniqueName: \"kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p\") pod \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") "
Nov 01 09:30:01.943112 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.943077    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca\") pod \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\" (UID: \"5af3f87b-9e47-4054-a99d-d4e7e113bab8\") "
Nov 01 09:30:01.943509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.943486    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca" pod "5af3f87b-9e47-4054-a99d-d4e7e113bab8" (UID: "5af3f87b-9e47-4054-a99d-d4e7e113bab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:01.944903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.944881    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca" pod "5af3f87b-9e47-4054-a99d-d4e7e113bab8" (UID: "5af3f87b-9e47-4054-a99d-d4e7e113bab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:30:01.945034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:01.945013    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p" pod "5af3f87b-9e47-4054-a99d-d4e7e113bab8" (UID: "5af3f87b-9e47-4054-a99d-d4e7e113bab8"). InnerVolumeSpecName "kube-api-access-4fk6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:02.045121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.045029    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fk6p\" (UniqueName: \"kubernetes.io/projected/5af3f87b-9e47-4054-a99d-d4e7e113bab8-kube-api-access-4fk6p\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:02.045121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.045070    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:02.045121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.045082    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af3f87b-9e47-4054-a99d-d4e7e113bab8-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:02.096957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.096902    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" podStartSLOduration=2.096887955 podStartE2EDuration="2.096887955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:02.094680818 +0000 UTC m=+3475.289075376" watchObservedRunningTime="2026-11-01 09:30:02.096887955 +0000 UTC m=+3475.291282509"
Nov 01 09:30:02.097340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.097317    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" podStartSLOduration=2.097311414 podStartE2EDuration="2.097311414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:01.984850048 +0000 UTC m=+3475.179244605" watchObservedRunningTime="2026-11-01 09:30:02.097311414 +0000 UTC m=+3475.291706017"
Nov 01 09:30:02.211930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.211871    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" podStartSLOduration=2.211852974 podStartE2EDuration="2.211852974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:02.205072435 +0000 UTC m=+3475.399466991" watchObservedRunningTime="2026-11-01 09:30:02.211852974 +0000 UTC m=+3475.406247530"
Nov 01 09:30:02.218951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.218928    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:02.437286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.437256    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-c4b11b"]
Nov 01 09:30:02.437918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.437903    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5af3f87b-9e47-4054-a99d-d4e7e113bab8" containerName="revoke-keys"
Nov 01 09:30:02.437960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.437919    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af3f87b-9e47-4054-a99d-d4e7e113bab8" containerName="revoke-keys"
Nov 01 09:30:02.438046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.438037    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af3f87b-9e47-4054-a99d-d4e7e113bab8" containerName="revoke-keys"
Nov 01 09:30:02.441644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.441618    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:02.490448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.490421    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-44wvn\""
Nov 01 09:30:02.491814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.491792    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-c4b11b"]
Nov 01 09:30:02.550501 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.550476    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwh8l\" (UniqueName: \"kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l\") pod \"test-curl-7415-c4b11b\" (UID: \"8a715bee-b344-4d96-9441-cec438e38e81\") " pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:02.646006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.645975    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2" event={"ID":"5af3f87b-9e47-4054-a99d-d4e7e113bab8","Type":"ContainerDied","Data":"11b159bf3c1b2869221fe5ecdaefe061bcfca2068832f0ec88f5752bcd7a9c25"}
Nov 01 09:30:02.646454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.646433    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11b159bf3c1b2869221fe5ecdaefe061bcfca2068832f0ec88f5752bcd7a9c25"
Nov 01 09:30:02.646555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.645986    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"
Nov 01 09:30:02.647620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.647598    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerStarted","Data":"27c40699e741e31c23fbfa7cf6e0da596a7da0c4f34163985e3331e8980fe687"}
Nov 01 09:30:02.649765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.649743    4314 generic.go:356] "Generic (PLEG): container finished" podID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerID="f7269cd59ed5e4e7d388ec6b0e3744522f419422f287315feb2a61f2735f3b66" exitCode=60
Nov 01 09:30:02.649851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.649797    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" event={"ID":"7b74d268-7fef-47fc-a995-fec1203c7e11","Type":"ContainerDied","Data":"f7269cd59ed5e4e7d388ec6b0e3744522f419422f287315feb2a61f2735f3b66"}
Nov 01 09:30:02.650077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.650064    4314 scope.go:122] "RemoveContainer" containerID="f7269cd59ed5e4e7d388ec6b0e3744522f419422f287315feb2a61f2735f3b66"
Nov 01 09:30:02.651179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.651162    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwh8l\" (UniqueName: \"kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l\") pod \"test-curl-7415-c4b11b\" (UID: \"8a715bee-b344-4d96-9441-cec438e38e81\") " pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:02.652352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.652334    4314 generic.go:356] "Generic (PLEG): container finished" podID="31989828-2bfe-49ac-a171-540ec24af1c0" containerID="18be8f5ba4bb77a4b63a417c6d5df48ccd70e2e6807fec2a35ecb74d13bf5201" exitCode=60
Nov 01 09:30:02.652425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.652389    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerDied","Data":"18be8f5ba4bb77a4b63a417c6d5df48ccd70e2e6807fec2a35ecb74d13bf5201"}
Nov 01 09:30:02.652656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.652642    4314 scope.go:122] "RemoveContainer" containerID="18be8f5ba4bb77a4b63a417c6d5df48ccd70e2e6807fec2a35ecb74d13bf5201"
Nov 01 09:30:02.654401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.654381    4314 generic.go:356] "Generic (PLEG): container finished" podID="4705630f-ed5a-4364-a36c-38278704ee92" containerID="69fc9cbdaf60e18ed89ea783e31bb5e5447cad647c8294063c986514d989c11f" exitCode=60
Nov 01 09:30:02.654495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.654455    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerDied","Data":"69fc9cbdaf60e18ed89ea783e31bb5e5447cad647c8294063c986514d989c11f"}
Nov 01 09:30:02.654815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.654646    4314 scope.go:122] "RemoveContainer" containerID="69fc9cbdaf60e18ed89ea783e31bb5e5447cad647c8294063c986514d989c11f"
Nov 01 09:30:02.656844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.656826    4314 generic.go:356] "Generic (PLEG): container finished" podID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerID="ef142fa3350a26da40f6fe8786b426a47b2fac37d28040a2076a44a3cf0117f9" exitCode=60
Nov 01 09:30:02.656933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.656887    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerDied","Data":"ef142fa3350a26da40f6fe8786b426a47b2fac37d28040a2076a44a3cf0117f9"}
Nov 01 09:30:02.657177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.657076    4314 scope.go:122] "RemoveContainer" containerID="ef142fa3350a26da40f6fe8786b426a47b2fac37d28040a2076a44a3cf0117f9"
Nov 01 09:30:02.681944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.681918    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 01 09:30:02.687797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.687607    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 01 09:30:02.688948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.688621    4314 generic.go:356] "Generic (PLEG): container finished" podID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerID="df7854cd33483f440563c0df0b331ddc0a911e70d5d63e1ae6192e03d9715183" exitCode=60
Nov 01 09:30:02.689628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.689046    4314 scope.go:122] "RemoveContainer" containerID="df7854cd33483f440563c0df0b331ddc0a911e70d5d63e1ae6192e03d9715183"
Nov 01 09:30:02.689628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.689079    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerDied","Data":"df7854cd33483f440563c0df0b331ddc0a911e70d5d63e1ae6192e03d9715183"}
Nov 01 09:30:02.692546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.692514    4314 generic.go:356] "Generic (PLEG): container finished" podID="aad05b42-95fc-4704-861d-ce903c61cff9" containerID="c8b0b4cb49f0dff6adb1ad62e7a8dbb43a952a3982d5383b40864ec93dcf28a2" exitCode=60
Nov 01 09:30:02.692652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.692564    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerDied","Data":"c8b0b4cb49f0dff6adb1ad62e7a8dbb43a952a3982d5383b40864ec93dcf28a2"}
Nov 01 09:30:02.692847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.692831    4314 scope.go:122] "RemoveContainer" containerID="c8b0b4cb49f0dff6adb1ad62e7a8dbb43a952a3982d5383b40864ec93dcf28a2"
Nov 01 09:30:02.694341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.694323    4314 generic.go:356] "Generic (PLEG): container finished" podID="5294f372-7697-4725-9642-3e189f9244a4" containerID="eaced095a90021c64906cc62a7997b18db421d49b668a4cef891b6dbb1087391" exitCode=60
Nov 01 09:30:02.694448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.694395    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" event={"ID":"5294f372-7697-4725-9642-3e189f9244a4","Type":"ContainerDied","Data":"eaced095a90021c64906cc62a7997b18db421d49b668a4cef891b6dbb1087391"}
Nov 01 09:30:02.694588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.694571    4314 scope.go:122] "RemoveContainer" containerID="eaced095a90021c64906cc62a7997b18db421d49b668a4cef891b6dbb1087391"
Nov 01 09:30:02.696247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.696226    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerStarted","Data":"0db66417e334a77824d69cd4003cffed38df37eec53280bf8489c88ae86833fe"}
Nov 01 09:30:02.697059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.697035    4314 scope.go:122] "RemoveContainer" containerID="0db66417e334a77824d69cd4003cffed38df37eec53280bf8489c88ae86833fe"
Nov 01 09:30:02.707742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.707723    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwh8l\" (UniqueName: \"kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l\") pod \"test-curl-7415-c4b11b\" (UID: \"8a715bee-b344-4d96-9441-cec438e38e81\") " pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:02.762792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.757885    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" podStartSLOduration=2.757866516 podStartE2EDuration="2.757866516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:02.757426521 +0000 UTC m=+3475.951821081" watchObservedRunningTime="2026-11-01 09:30:02.757866516 +0000 UTC m=+3475.952261071"
Nov 01 09:30:02.840026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:02.839962    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:03.297867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.297841    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-c4b11b"]
Nov 01 09:30:03.369971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.369892    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:03.680951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.680870    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 01 09:30:03.681487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.681456    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 01 09:30:03.746517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.746480    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerStarted","Data":"796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1"}
Nov 01 09:30:03.749133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.749102    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerStarted","Data":"445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e"}
Nov 01 09:30:03.751610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.751585    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerStarted","Data":"59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590"}
Nov 01 09:30:03.755061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.755036    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerStarted","Data":"ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd"}
Nov 01 09:30:03.791287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.791244    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" event={"ID":"5294f372-7697-4725-9642-3e189f9244a4","Type":"ContainerStarted","Data":"c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7"}
Nov 01 09:30:03.803151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.803120    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-c4b11b" event={"ID":"8a715bee-b344-4d96-9441-cec438e38e81","Type":"ContainerStarted","Data":"47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37"}
Nov 01 09:30:03.803354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.803338    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-c4b11b" event={"ID":"8a715bee-b344-4d96-9441-cec438e38e81","Type":"ContainerStarted","Data":"1de4feb06255984d224469835a0415720a5e7fe1292e3bc52f6d767e4f1314bf"}
Nov 01 09:30:03.806013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.805991    4314 generic.go:356] "Generic (PLEG): container finished" podID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerID="0db66417e334a77824d69cd4003cffed38df37eec53280bf8489c88ae86833fe" exitCode=60
Nov 01 09:30:03.806150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.806129    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerDied","Data":"0db66417e334a77824d69cd4003cffed38df37eec53280bf8489c88ae86833fe"}
Nov 01 09:30:03.806248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.806181    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerStarted","Data":"aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa"}
Nov 01 09:30:03.808504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.808480    4314 generic.go:356] "Generic (PLEG): container finished" podID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerID="27c40699e741e31c23fbfa7cf6e0da596a7da0c4f34163985e3331e8980fe687" exitCode=60
Nov 01 09:30:03.808625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.808570    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerDied","Data":"27c40699e741e31c23fbfa7cf6e0da596a7da0c4f34163985e3331e8980fe687"}
Nov 01 09:30:03.808868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.808851    4314 scope.go:122] "RemoveContainer" containerID="27c40699e741e31c23fbfa7cf6e0da596a7da0c4f34163985e3331e8980fe687"
Nov 01 09:30:03.811581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.811508    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" event={"ID":"7b74d268-7fef-47fc-a995-fec1203c7e11","Type":"ContainerStarted","Data":"3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4"}
Nov 01 09:30:03.813748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.813724    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerStarted","Data":"e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"}
Nov 01 09:30:03.814077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.814061    4314 scope.go:122] "RemoveContainer" containerID="e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"
Nov 01 09:30:03.814797 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:03.814774    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k_odh-ai-gateway-infra(31989828-2bfe-49ac-a171-540ec24af1c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" podUID="31989828-2bfe-49ac-a171-540ec24af1c0"
Nov 01 09:30:03.988568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:03.984753    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" podStartSLOduration=3.984730454 podStartE2EDuration="3.984730454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:03.98339792 +0000 UTC m=+3477.177792475" watchObservedRunningTime="2026-11-01 09:30:03.984730454 +0000 UTC m=+3477.179125009"
Nov 01 09:30:04.209105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.209046    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" podStartSLOduration=4.209026731 podStartE2EDuration="4.209026731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:04.202616307 +0000 UTC m=+3477.397010865" watchObservedRunningTime="2026-11-01 09:30:04.209026731 +0000 UTC m=+3477.403421286"
Nov 01 09:30:04.430299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.430246    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" podStartSLOduration=4.430232388 podStartE2EDuration="4.430232388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:04.42973883 +0000 UTC m=+3477.624133389" watchObservedRunningTime="2026-11-01 09:30:04.430232388 +0000 UTC m=+3477.624626944"
Nov 01 09:30:04.679283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.679253    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 01 09:30:04.679523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.679496    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 01 09:30:04.834584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.834483    4314 generic.go:356] "Generic (PLEG): container finished" podID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerID="aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa" exitCode=60
Nov 01 09:30:04.835042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.834642    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerDied","Data":"aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa"}
Nov 01 09:30:04.835042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.834718    4314 scope.go:122] "RemoveContainer" containerID="0db66417e334a77824d69cd4003cffed38df37eec53280bf8489c88ae86833fe"
Nov 01 09:30:04.835042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.834988    4314 scope.go:122] "RemoveContainer" containerID="aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa"
Nov 01 09:30:04.835321 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.835299    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk_odh-ai-gateway-infra(8aa0d36e-7de3-4603-8604-ee0f06ddadf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1"
Nov 01 09:30:04.838000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.837972    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerStarted","Data":"1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a"}
Nov 01 09:30:04.840271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.840235    4314 generic.go:356] "Generic (PLEG): container finished" podID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerID="3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4" exitCode=60
Nov 01 09:30:04.840353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.840313    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" event={"ID":"7b74d268-7fef-47fc-a995-fec1203c7e11","Type":"ContainerDied","Data":"3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4"}
Nov 01 09:30:04.841260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.841244    4314 scope.go:122] "RemoveContainer" containerID="3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4"
Nov 01 09:30:04.841514 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.841486    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh_odh-ai-gateway-infra(7b74d268-7fef-47fc-a995-fec1203c7e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11"
Nov 01 09:30:04.842903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.842882    4314 generic.go:356] "Generic (PLEG): container finished" podID="31989828-2bfe-49ac-a171-540ec24af1c0" containerID="e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441" exitCode=60
Nov 01 09:30:04.842978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.842966    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerDied","Data":"e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"}
Nov 01 09:30:04.843227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.843209    4314 scope.go:122] "RemoveContainer" containerID="e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"
Nov 01 09:30:04.843480 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.843458    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k_odh-ai-gateway-infra(31989828-2bfe-49ac-a171-540ec24af1c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" podUID="31989828-2bfe-49ac-a171-540ec24af1c0"
Nov 01 09:30:04.844920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.844900    4314 generic.go:356] "Generic (PLEG): container finished" podID="4705630f-ed5a-4364-a36c-38278704ee92" containerID="796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1" exitCode=60
Nov 01 09:30:04.845035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.844956    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerDied","Data":"796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1"}
Nov 01 09:30:04.845190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.845176    4314 scope.go:122] "RemoveContainer" containerID="796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1"
Nov 01 09:30:04.845369 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.845354    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72_odh-ai-gateway-infra(4705630f-ed5a-4364-a36c-38278704ee92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" podUID="4705630f-ed5a-4364-a36c-38278704ee92"
Nov 01 09:30:04.847567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.847190    4314 generic.go:356] "Generic (PLEG): container finished" podID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerID="445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e" exitCode=60
Nov 01 09:30:04.847567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.847227    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerDied","Data":"445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e"}
Nov 01 09:30:04.847567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.847449    4314 scope.go:122] "RemoveContainer" containerID="445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e"
Nov 01 09:30:04.847778 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.847699    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp_odh-ai-gateway-infra(b76c0b14-cede-4b3a-80e0-cdc8beaf9724)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724"
Nov 01 09:30:04.849260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.849232    4314 generic.go:356] "Generic (PLEG): container finished" podID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerID="59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590" exitCode=60
Nov 01 09:30:04.849349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.849316    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerDied","Data":"59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590"}
Nov 01 09:30:04.849508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.849491    4314 scope.go:122] "RemoveContainer" containerID="59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590"
Nov 01 09:30:04.849780 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.849757    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5_odh-ai-gateway-infra(8849cd98-018b-4ff4-a9a0-02cf3ce8d86a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a"
Nov 01 09:30:04.851257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.851229    4314 generic.go:356] "Generic (PLEG): container finished" podID="aad05b42-95fc-4704-861d-ce903c61cff9" containerID="ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd" exitCode=60
Nov 01 09:30:04.851353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.851311    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerDied","Data":"ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd"}
Nov 01 09:30:04.851482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.851466    4314 scope.go:122] "RemoveContainer" containerID="ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd"
Nov 01 09:30:04.851672 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.851653    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd_odh-ai-gateway-infra(aad05b42-95fc-4704-861d-ce903c61cff9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" podUID="aad05b42-95fc-4704-861d-ce903c61cff9"
Nov 01 09:30:04.854696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.854676    4314 scope.go:122] "RemoveContainer" containerID="f7269cd59ed5e4e7d388ec6b0e3744522f419422f287315feb2a61f2735f3b66"
Nov 01 09:30:04.855126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.855107    4314 generic.go:356] "Generic (PLEG): container finished" podID="5294f372-7697-4725-9642-3e189f9244a4" containerID="c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7" exitCode=60
Nov 01 09:30:04.855234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.855215    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" event={"ID":"5294f372-7697-4725-9642-3e189f9244a4","Type":"ContainerDied","Data":"c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7"}
Nov 01 09:30:04.855582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.855527    4314 scope.go:122] "RemoveContainer" containerID="c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7"
Nov 01 09:30:04.855937 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:04.855902    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6_odh-ai-gateway-infra(5294f372-7697-4725-9642-3e189f9244a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" podUID="5294f372-7697-4725-9642-3e189f9244a4"
Nov 01 09:30:04.884330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.883813    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7415-c4b11b" podStartSLOduration=2.883794561 podStartE2EDuration="2.883794561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:04.85489967 +0000 UTC m=+3478.049294226" watchObservedRunningTime="2026-11-01 09:30:04.883794561 +0000 UTC m=+3478.078189117"
Nov 01 09:30:04.896589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.896565    4314 scope.go:122] "RemoveContainer" containerID="18be8f5ba4bb77a4b63a417c6d5df48ccd70e2e6807fec2a35ecb74d13bf5201"
Nov 01 09:30:04.909709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.909688    4314 scope.go:122] "RemoveContainer" containerID="69fc9cbdaf60e18ed89ea783e31bb5e5447cad647c8294063c986514d989c11f"
Nov 01 09:30:04.922711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.922691    4314 scope.go:122] "RemoveContainer" containerID="ef142fa3350a26da40f6fe8786b426a47b2fac37d28040a2076a44a3cf0117f9"
Nov 01 09:30:04.939180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.938812    4314 scope.go:122] "RemoveContainer" containerID="df7854cd33483f440563c0df0b331ddc0a911e70d5d63e1ae6192e03d9715183"
Nov 01 09:30:04.952731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.952713    4314 scope.go:122] "RemoveContainer" containerID="c8b0b4cb49f0dff6adb1ad62e7a8dbb43a952a3982d5383b40864ec93dcf28a2"
Nov 01 09:30:04.999823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:04.999802    4314 scope.go:122] "RemoveContainer" containerID="eaced095a90021c64906cc62a7997b18db421d49b668a4cef891b6dbb1087391"
Nov 01 09:30:05.679612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.678883    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 01 09:30:05.679612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.678948    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 01 09:30:05.892759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.892728    4314 scope.go:122] "RemoveContainer" containerID="aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa"
Nov 01 09:30:05.893273 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.893103    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk_odh-ai-gateway-infra(8aa0d36e-7de3-4603-8604-ee0f06ddadf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1"
Nov 01 09:30:05.895001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.894975    4314 generic.go:356] "Generic (PLEG): container finished" podID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerID="1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a" exitCode=60
Nov 01 09:30:05.895164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.895051    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerDied","Data":"1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a"}
Nov 01 09:30:05.895164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.895139    4314 scope.go:122] "RemoveContainer" containerID="27c40699e741e31c23fbfa7cf6e0da596a7da0c4f34163985e3331e8980fe687"
Nov 01 09:30:05.895314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.895300    4314 scope.go:122] "RemoveContainer" containerID="1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a"
Nov 01 09:30:05.895631 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.895608    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff_odh-ai-gateway-infra(e434ec4a-df75-4ae0-89bc-a20ff6c3d726)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726"
Nov 01 09:30:05.898515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.898494    4314 scope.go:122] "RemoveContainer" containerID="3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4"
Nov 01 09:30:05.898821 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.898775    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh_odh-ai-gateway-infra(7b74d268-7fef-47fc-a995-fec1203c7e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11"
Nov 01 09:30:05.900353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.900335    4314 scope.go:122] "RemoveContainer" containerID="e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"
Nov 01 09:30:05.900562 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.900523    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k_odh-ai-gateway-infra(31989828-2bfe-49ac-a171-540ec24af1c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" podUID="31989828-2bfe-49ac-a171-540ec24af1c0"
Nov 01 09:30:05.901990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.901974    4314 scope.go:122] "RemoveContainer" containerID="796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1"
Nov 01 09:30:05.902182 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.902164    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72_odh-ai-gateway-infra(4705630f-ed5a-4364-a36c-38278704ee92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" podUID="4705630f-ed5a-4364-a36c-38278704ee92"
Nov 01 09:30:05.903676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.903655    4314 scope.go:122] "RemoveContainer" containerID="445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e"
Nov 01 09:30:05.903882 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.903858    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp_odh-ai-gateway-infra(b76c0b14-cede-4b3a-80e0-cdc8beaf9724)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724"
Nov 01 09:30:05.905240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.905224    4314 scope.go:122] "RemoveContainer" containerID="59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590"
Nov 01 09:30:05.905434 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.905415    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5_odh-ai-gateway-infra(8849cd98-018b-4ff4-a9a0-02cf3ce8d86a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a"
Nov 01 09:30:05.906873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.906855    4314 scope.go:122] "RemoveContainer" containerID="ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd"
Nov 01 09:30:05.907088 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.907062    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd_odh-ai-gateway-infra(aad05b42-95fc-4704-861d-ce903c61cff9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" podUID="aad05b42-95fc-4704-861d-ce903c61cff9"
Nov 01 09:30:05.908469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:05.908454    4314 scope.go:122] "RemoveContainer" containerID="c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7"
Nov 01 09:30:05.908686 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:05.908666    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6_odh-ai-gateway-infra(5294f372-7697-4725-9642-3e189f9244a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" podUID="5294f372-7697-4725-9642-3e189f9244a4"
Nov 01 09:30:06.300201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.300166    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-c4b11b"]
Nov 01 09:30:06.300395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.300362    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7415-c4b11b" podUID="8a715bee-b344-4d96-9441-cec438e38e81" containerName="test-curl-7415-c4b11b" containerID="cri-o://47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37" gracePeriod=2
Nov 01 09:30:06.304087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.304061    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-c4b11b"]
Nov 01 09:30:06.451092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.451061    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-2e62c8"]
Nov 01 09:30:06.451710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.451695    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8a715bee-b344-4d96-9441-cec438e38e81" containerName="test-curl-7415-c4b11b"
Nov 01 09:30:06.451779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.451712    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a715bee-b344-4d96-9441-cec438e38e81" containerName="test-curl-7415-c4b11b"
Nov 01 09:30:06.451877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.451868    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a715bee-b344-4d96-9441-cec438e38e81" containerName="test-curl-7415-c4b11b"
Nov 01 09:30:06.456625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.456610    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:06.508246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.508207    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-2e62c8"]
Nov 01 09:30:06.534475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.534442    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrdm\" (UniqueName: \"kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm\") pod \"test-curl-7415-2e62c8\" (UID: \"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d\") " pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:06.635638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.635525    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrdm\" (UniqueName: \"kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm\") pod \"test-curl-7415-2e62c8\" (UID: \"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d\") " pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:06.679219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.679183    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 01 09:30:06.679352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.679243    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 01 09:30:06.692666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.692639    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrdm\" (UniqueName: \"kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm\") pod \"test-curl-7415-2e62c8\" (UID: \"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d\") " pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:06.769051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.769025    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:06.856574 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:06.854037    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-c4b11b\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="8a715bee-b344-4d96-9441-cec438e38e81" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:06.919659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:06.919571    4314 scope.go:122] "RemoveContainer" containerID="1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a"
Nov 01 09:30:06.920164 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:06.919905    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff_odh-ai-gateway-infra(e434ec4a-df75-4ae0-89bc-a20ff6c3d726)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726"
Nov 01 09:30:07.145119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:07.145087    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-2e62c8"]
Nov 01 09:30:07.145973 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:07.145945    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65a5c21c_fbdd_47a0_bb3d_9090b7095f6d.slice/crio-7dbf65d3793c9e07879984f272fb2b9ba78ad51386de1c3b84c72be6291e65d1 WatchSource:0}: Error finding container 7dbf65d3793c9e07879984f272fb2b9ba78ad51386de1c3b84c72be6291e65d1: Status 404 returned error can't find the container with id 7dbf65d3793c9e07879984f272fb2b9ba78ad51386de1c3b84c72be6291e65d1
Nov 01 09:30:07.683626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:07.683596    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:30:07.684022 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:07.684003    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:30:07.684711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:07.684689    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:30:07.933099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:07.933065    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-2e62c8" event={"ID":"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d","Type":"ContainerStarted","Data":"3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38"}
Nov 01 09:30:07.933493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:07.933108    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-2e62c8" event={"ID":"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d","Type":"ContainerStarted","Data":"7dbf65d3793c9e07879984f272fb2b9ba78ad51386de1c3b84c72be6291e65d1"}
Nov 01 09:30:08.041228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.041169    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7415-2e62c8" podStartSLOduration=2.04115555 podStartE2EDuration="2.04115555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:08.037813319 +0000 UTC m=+3481.232207875" watchObservedRunningTime="2026-11-01 09:30:08.04115555 +0000 UTC m=+3481.235550104"
Nov 01 09:30:08.124325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.124298    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:08.286528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.286493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:08.428342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.428302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:08.545876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.545850    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:08.593307 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:08.593243    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-c4b11b\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="8a715bee-b344-4d96-9441-cec438e38e81" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:08.660999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.660970    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l\" (UniqueName: \"kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l\") pod \"8a715bee-b344-4d96-9441-cec438e38e81\" (UID: \"8a715bee-b344-4d96-9441-cec438e38e81\") "
Nov 01 09:30:08.663108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.663077    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l" pod "8a715bee-b344-4d96-9441-cec438e38e81" (UID: "8a715bee-b344-4d96-9441-cec438e38e81"). InnerVolumeSpecName "kube-api-access-jwh8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:08.762984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.762942    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwh8l\" (UniqueName: \"kubernetes.io/projected/8a715bee-b344-4d96-9441-cec438e38e81-kube-api-access-jwh8l\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:08.929119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.929043    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a715bee-b344-4d96-9441-cec438e38e81" path="/var/lib/kubelet/pods/8a715bee-b344-4d96-9441-cec438e38e81/volumes"
Nov 01 09:30:08.939353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.939330    4314 generic.go:356] "Generic (PLEG): container finished" podID="8a715bee-b344-4d96-9441-cec438e38e81" containerID="47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37" exitCode=137
Nov 01 09:30:08.939791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.939373    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:08.939791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.939422    4314 scope.go:122] "RemoveContainer" containerID="47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37"
Nov 01 09:30:08.948799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.948758    4314 scope.go:122] "RemoveContainer" containerID="47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37"
Nov 01 09:30:08.949063 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:08.949044    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37\": container with ID starting with 47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37 not found: ID does not exist" containerID="47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37"
Nov 01 09:30:08.949126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:08.949070    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37"} err="failed to get container status \"47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37\": rpc error: code = NotFound desc = could not find container \"47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37\": container with ID starting with 47f7c78a96dbfe9818a8f1ffae9e1b1d947033246f3e2aa79f73bb3d97512e37 not found: ID does not exist"
Nov 01 09:30:08.986888 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:08.986858    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-c4b11b\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="8a715bee-b344-4d96-9441-cec438e38e81" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:09.039490 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:09.039453    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-c4b11b\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="8a715bee-b344-4d96-9441-cec438e38e81" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:09.579118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.579019    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-2e62c8"]
Nov 01 09:30:09.584392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.584365    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-2e62c8"]
Nov 01 09:30:09.721119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.721087    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-10423f"]
Nov 01 09:30:09.721774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.721760    4314 cpu_manager.go:414] "containerMap: removing container" podUID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" containerName="test-curl-7415-2e62c8"
Nov 01 09:30:09.721816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.721776    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" containerName="test-curl-7415-2e62c8"
Nov 01 09:30:09.721884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.721875    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" containerName="test-curl-7415-2e62c8"
Nov 01 09:30:09.726873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.726859    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:09.776565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.776528    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-10423f"]
Nov 01 09:30:09.877259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.877186    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqxsd\" (UniqueName: \"kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd\") pod \"test-curl-7415-10423f\" (UID: \"2ec35e67-f606-43e3-8936-8c1df6919b47\") " pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:09.877665 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:09.877633    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-c4b11b\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="8a715bee-b344-4d96-9441-cec438e38e81" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:09.946301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.946270    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7415-2e62c8" podUID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" containerName="test-curl-7415-2e62c8" containerID="cri-o://3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38" gracePeriod=2
Nov 01 09:30:09.978885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:09.978848    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqxsd\" (UniqueName: \"kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd\") pod \"test-curl-7415-10423f\" (UID: \"2ec35e67-f606-43e3-8936-8c1df6919b47\") " pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:09.993704 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:09.993679    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-2e62c8\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:10.035817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:10.035790    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqxsd\" (UniqueName: \"kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd\") pod \"test-curl-7415-10423f\" (UID: \"2ec35e67-f606-43e3-8936-8c1df6919b47\") " pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:10.039659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:10.039640    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:10.040829 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:10.040798    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-c4b11b\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="8a715bee-b344-4d96-9441-cec438e38e81" pod="opendatahub/test-curl-7415-c4b11b"
Nov 01 09:30:10.411321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:10.411291    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-10423f"]
Nov 01 09:30:10.412449 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:10.412424    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec35e67_f606_43e3_8936_8c1df6919b47.slice/crio-5bd1fa4e1b03385b7eccb98285ac648933d8961a7f5c73e9a2be7266f8669072 WatchSource:0}: Error finding container 5bd1fa4e1b03385b7eccb98285ac648933d8961a7f5c73e9a2be7266f8669072: Status 404 returned error can't find the container with id 5bd1fa4e1b03385b7eccb98285ac648933d8961a7f5c73e9a2be7266f8669072
Nov 01 09:30:10.954297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:10.954266    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-10423f" event={"ID":"2ec35e67-f606-43e3-8936-8c1df6919b47","Type":"ContainerStarted","Data":"dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9"}
Nov 01 09:30:10.954762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:10.954306    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-10423f" event={"ID":"2ec35e67-f606-43e3-8936-8c1df6919b47","Type":"ContainerStarted","Data":"5bd1fa4e1b03385b7eccb98285ac648933d8961a7f5c73e9a2be7266f8669072"}
Nov 01 09:30:11.062289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:11.062240    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7415-10423f" podStartSLOduration=2.062225329 podStartE2EDuration="2.062225329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:11.061609057 +0000 UTC m=+3484.256003617" watchObservedRunningTime="2026-11-01 09:30:11.062225329 +0000 UTC m=+3484.256619886"
Nov 01 09:30:11.323672 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:11.323634    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:30:12.221499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.221474    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:12.304816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.304788    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm\" (UniqueName: \"kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm\") pod \"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d\" (UID: \"65a5c21c-fbdd-47a0-bb3d-9090b7095f6d\") "
Nov 01 09:30:12.306851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.306821    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm" pod "65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" (UID: "65a5c21c-fbdd-47a0-bb3d-9090b7095f6d"). InnerVolumeSpecName "kube-api-access-7zrdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:12.406446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.406416    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zrdm\" (UniqueName: \"kubernetes.io/projected/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d-kube-api-access-7zrdm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.464868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.464802    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-10423f"]
Nov 01 09:30:12.471736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.471709    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-10423f"]
Nov 01 09:30:12.929800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.929766    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" path="/var/lib/kubelet/pods/65a5c21c-fbdd-47a0-bb3d-9090b7095f6d/volumes"
Nov 01 09:30:12.972147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.972121    4314 generic.go:356] "Generic (PLEG): container finished" podID="65a5c21c-fbdd-47a0-bb3d-9090b7095f6d" containerID="3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38" exitCode=137
Nov 01 09:30:12.972271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.972158    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-2e62c8"
Nov 01 09:30:12.972271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.972214    4314 scope.go:122] "RemoveContainer" containerID="3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38"
Nov 01 09:30:12.972446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.972421    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7415-10423f" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" containerName="test-curl-7415-10423f" containerID="cri-o://dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9" gracePeriod=2
Nov 01 09:30:12.982958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.982928    4314 scope.go:122] "RemoveContainer" containerID="3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38"
Nov 01 09:30:12.983287 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:12.983256    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38\": container with ID starting with 3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38 not found: ID does not exist" containerID="3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38"
Nov 01 09:30:12.983402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:12.983299    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38"} err="failed to get container status \"3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38\": rpc error: code = NotFound desc = could not find container \"3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38\": container with ID starting with 3333b6a0cccc3ee1d9aea964ac0addb5a080f0162fa439231e54a5aaa2d9ae38 not found: ID does not exist"
Nov 01 09:30:13.187556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:13.187453    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:13.942435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:13.942400    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:14.084490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:14.084459    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:14.351679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:14.351644    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:14.921506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:14.921473    4314 scope.go:122] "RemoveContainer" containerID="aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa"
Nov 01 09:30:14.921708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:14.921637    4314 scope.go:122] "RemoveContainer" containerID="59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590"
Nov 01 09:30:14.921925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:14.921899    4314 scope.go:122] "RemoveContainer" containerID="3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4"
Nov 01 09:30:15.304767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.304743    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:15.446264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.446183    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd\" (UniqueName: \"kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd\") pod \"2ec35e67-f606-43e3-8936-8c1df6919b47\" (UID: \"2ec35e67-f606-43e3-8936-8c1df6919b47\") "
Nov 01 09:30:15.448089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.448064    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd" pod "2ec35e67-f606-43e3-8936-8c1df6919b47" (UID: "2ec35e67-f606-43e3-8936-8c1df6919b47"). InnerVolumeSpecName "kube-api-access-rqxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:15.547415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.547387    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqxsd\" (UniqueName: \"kubernetes.io/projected/2ec35e67-f606-43e3-8936-8c1df6919b47-kube-api-access-rqxsd\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:15.921976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.921944    4314 scope.go:122] "RemoveContainer" containerID="c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7"
Nov 01 09:30:15.922137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.922118    4314 scope.go:122] "RemoveContainer" containerID="ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd"
Nov 01 09:30:15.999224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.999197    4314 generic.go:356] "Generic (PLEG): container finished" podID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerID="b6df1604ecdcb74a2008079fc93c4ff2e7195003cc5b3071c53beda6ac2a0bb2" exitCode=60
Nov 01 09:30:15.999334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.999228    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerDied","Data":"b6df1604ecdcb74a2008079fc93c4ff2e7195003cc5b3071c53beda6ac2a0bb2"}
Nov 01 09:30:15.999334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.999287    4314 scope.go:122] "RemoveContainer" containerID="aa6fe91c23c0a8f8879e7cd452b8fcb6c8062b8cc42c70272b87040951dcfefa"
Nov 01 09:30:15.999707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:15.999690    4314 scope.go:122] "RemoveContainer" containerID="b6df1604ecdcb74a2008079fc93c4ff2e7195003cc5b3071c53beda6ac2a0bb2"
Nov 01 09:30:16.000020 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:15.999998    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk_odh-ai-gateway-infra(8aa0d36e-7de3-4603-8604-ee0f06ddadf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1"
Nov 01 09:30:16.001681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.001656    4314 generic.go:356] "Generic (PLEG): container finished" podID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerID="8216be9f86dea359177393cd7aaa892a3723ceb9a4ca2267c2e46b30b29567bc" exitCode=60
Nov 01 09:30:16.001781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.001714    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" event={"ID":"7b74d268-7fef-47fc-a995-fec1203c7e11","Type":"ContainerDied","Data":"8216be9f86dea359177393cd7aaa892a3723ceb9a4ca2267c2e46b30b29567bc"}
Nov 01 09:30:16.002047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.002031    4314 scope.go:122] "RemoveContainer" containerID="8216be9f86dea359177393cd7aaa892a3723ceb9a4ca2267c2e46b30b29567bc"
Nov 01 09:30:16.002320 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:16.002298    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh_odh-ai-gateway-infra(7b74d268-7fef-47fc-a995-fec1203c7e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11"
Nov 01 09:30:16.003671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.003654    4314 generic.go:356] "Generic (PLEG): container finished" podID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerID="2a3685653267f08698237d8364632bfb54fd3af1430f4df8094dc7c93de96710" exitCode=60
Nov 01 09:30:16.003765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.003727    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerDied","Data":"2a3685653267f08698237d8364632bfb54fd3af1430f4df8094dc7c93de96710"}
Nov 01 09:30:16.004041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.004025    4314 scope.go:122] "RemoveContainer" containerID="2a3685653267f08698237d8364632bfb54fd3af1430f4df8094dc7c93de96710"
Nov 01 09:30:16.004279 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:16.004258    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5_odh-ai-gateway-infra(8849cd98-018b-4ff4-a9a0-02cf3ce8d86a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a"
Nov 01 09:30:16.005108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.005087    4314 generic.go:356] "Generic (PLEG): container finished" podID="2ec35e67-f606-43e3-8936-8c1df6919b47" containerID="dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9" exitCode=137
Nov 01 09:30:16.005195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.005131    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:16.021129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.021105    4314 scope.go:122] "RemoveContainer" containerID="3ab0809d801cf051a18a8f752149d701bebd4e10c4f2e654574f3fc7aa2c1ae4"
Nov 01 09:30:16.041980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.041964    4314 scope.go:122] "RemoveContainer" containerID="59cbdef076ac7a7ccc55e4dc29044834aaf6b253e1cb26013df34c66d5ebd590"
Nov 01 09:30:16.054960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.054944    4314 scope.go:122] "RemoveContainer" containerID="dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9"
Nov 01 09:30:16.064458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.064428    4314 scope.go:122] "RemoveContainer" containerID="dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9"
Nov 01 09:30:16.064706 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:16.064686    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9\": container with ID starting with dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9 not found: ID does not exist" containerID="dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9"
Nov 01 09:30:16.064767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.064714    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9"} err="failed to get container status \"dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9\": rpc error: code = NotFound desc = could not find container \"dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9\": container with ID starting with dbcc7b424d77ac42f2db5dea3e07aad8d16a3304d3371f7e63e3c07cc161b0b9 not found: ID does not exist"
Nov 01 09:30:16.260474 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:16.260438    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-10423f\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:16.926715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.926691    4314 scope.go:122] "RemoveContainer" containerID="e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"
Nov 01 09:30:16.927032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.926818    4314 scope.go:122] "RemoveContainer" containerID="445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e"
Nov 01 09:30:16.927032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.926853    4314 scope.go:122] "RemoveContainer" containerID="796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1"
Nov 01 09:30:16.931017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:16.930995    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" path="/var/lib/kubelet/pods/2ec35e67-f606-43e3-8936-8c1df6919b47/volumes"
Nov 01 09:30:16.973619 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:16.973582    4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-10423f\" is forbidden: User \"system:node:ip-10-0-134-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-93.us-west-2.compute.internal' and this object" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" pod="opendatahub/test-curl-7415-10423f"
Nov 01 09:30:17.015846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.015770    4314 generic.go:356] "Generic (PLEG): container finished" podID="aad05b42-95fc-4704-861d-ce903c61cff9" containerID="3eec99721827f4aeff02a93441264fc3bbfe1c3a48b4656b8ad7a48a1142facc" exitCode=60
Nov 01 09:30:17.015997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.015853    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerDied","Data":"3eec99721827f4aeff02a93441264fc3bbfe1c3a48b4656b8ad7a48a1142facc"}
Nov 01 09:30:17.015997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.015920    4314 scope.go:122] "RemoveContainer" containerID="ef9c21b58931c8aacff6b74e91854d116bbdddbef5316331d527806f35add2fd"
Nov 01 09:30:17.016558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.016330    4314 scope.go:122] "RemoveContainer" containerID="3eec99721827f4aeff02a93441264fc3bbfe1c3a48b4656b8ad7a48a1142facc"
Nov 01 09:30:17.016706 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:17.016684    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd_odh-ai-gateway-infra(aad05b42-95fc-4704-861d-ce903c61cff9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" podUID="aad05b42-95fc-4704-861d-ce903c61cff9"
Nov 01 09:30:17.017846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.017824    4314 generic.go:356] "Generic (PLEG): container finished" podID="5294f372-7697-4725-9642-3e189f9244a4" containerID="d5d778b2a1126c5015d14c733c52814fd5d47c184d2e39d59d8ec4c8b7302e5a" exitCode=60
Nov 01 09:30:17.017922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.017885    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" event={"ID":"5294f372-7697-4725-9642-3e189f9244a4","Type":"ContainerDied","Data":"d5d778b2a1126c5015d14c733c52814fd5d47c184d2e39d59d8ec4c8b7302e5a"}
Nov 01 09:30:17.018250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.018234    4314 scope.go:122] "RemoveContainer" containerID="d5d778b2a1126c5015d14c733c52814fd5d47c184d2e39d59d8ec4c8b7302e5a"
Nov 01 09:30:17.018525 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:17.018502    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6_odh-ai-gateway-infra(5294f372-7697-4725-9642-3e189f9244a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" podUID="5294f372-7697-4725-9642-3e189f9244a4"
Nov 01 09:30:17.030166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.030149    4314 scope.go:122] "RemoveContainer" containerID="c6861166c377e512390107d7babbc127a00ffc130131e55655edb6674d9eb9a7"
Nov 01 09:30:17.123136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.123103    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"]
Nov 01 09:30:17.226104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.226069    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"]
Nov 01 09:30:17.381870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.381846    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"]
Nov 01 09:30:17.397709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.397685    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:17.402040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.401997    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:17.583214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583190    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:17.583434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583412    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca\") pod \"7b74d268-7fef-47fc-a995-fec1203c7e11\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") "
Nov 01 09:30:17.583560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583516    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca\") pod \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") "
Nov 01 09:30:17.583663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583649    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r\" (UniqueName: \"kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r\") pod \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\" (UID: \"8aa0d36e-7de3-4603-8604-ee0f06ddadf1\") "
Nov 01 09:30:17.583744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583729    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj\" (UniqueName: \"kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj\") pod \"7b74d268-7fef-47fc-a995-fec1203c7e11\" (UID: \"7b74d268-7fef-47fc-a995-fec1203c7e11\") "
Nov 01 09:30:17.583853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583830    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca" pod "7b74d268-7fef-47fc-a995-fec1203c7e11" (UID: "7b74d268-7fef-47fc-a995-fec1203c7e11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:17.583914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.583837    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca" pod "8aa0d36e-7de3-4603-8604-ee0f06ddadf1" (UID: "8aa0d36e-7de3-4603-8604-ee0f06ddadf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:17.584217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.584196    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74d268-7fef-47fc-a995-fec1203c7e11-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.584326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.584220    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.589646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.589623    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj" pod "7b74d268-7fef-47fc-a995-fec1203c7e11" (UID: "7b74d268-7fef-47fc-a995-fec1203c7e11"). InnerVolumeSpecName "kube-api-access-j7xwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.589759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.589685    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r" pod "8aa0d36e-7de3-4603-8604-ee0f06ddadf1" (UID: "8aa0d36e-7de3-4603-8604-ee0f06ddadf1"). InnerVolumeSpecName "kube-api-access-7rv9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.684704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.684673    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca\") pod \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") "
Nov 01 09:30:17.684893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.684830    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb\" (UniqueName: \"kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb\") pod \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\" (UID: \"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a\") "
Nov 01 09:30:17.685054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.685032    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca" pod "8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" (UID: "8849cd98-018b-4ff4-a9a0-02cf3ce8d86a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:17.685476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.685447    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rv9r\" (UniqueName: \"kubernetes.io/projected/8aa0d36e-7de3-4603-8604-ee0f06ddadf1-kube-api-access-7rv9r\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.685624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.685482    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.685624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.685498    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7xwj\" (UniqueName: \"kubernetes.io/projected/7b74d268-7fef-47fc-a995-fec1203c7e11-kube-api-access-j7xwj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.687755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.687726    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb" pod "8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" (UID: "8849cd98-018b-4ff4-a9a0-02cf3ce8d86a"). InnerVolumeSpecName "kube-api-access-l9bzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.787314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.787227    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bzb\" (UniqueName: \"kubernetes.io/projected/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a-kube-api-access-l9bzb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.989413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:17.989384    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:18.029597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.029572    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"
Nov 01 09:30:18.029597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.029574    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh" event={"ID":"7b74d268-7fef-47fc-a995-fec1203c7e11","Type":"ContainerDied","Data":"ede86aa921fcef620db3860f9930587f1b37b506e3bf1d926f0c45921c59a577"}
Nov 01 09:30:18.030977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.030761    4314 scope.go:122] "RemoveContainer" containerID="8216be9f86dea359177393cd7aaa892a3723ceb9a4ca2267c2e46b30b29567bc"
Nov 01 09:30:18.035687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.035474    4314 generic.go:356] "Generic (PLEG): container finished" podID="31989828-2bfe-49ac-a171-540ec24af1c0" containerID="7bbf8f2f686fd3fc3157bfe8107dc1007f6fcadba46cf51074431305968f35c2" exitCode=60
Nov 01 09:30:18.035687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.035558    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerDied","Data":"7bbf8f2f686fd3fc3157bfe8107dc1007f6fcadba46cf51074431305968f35c2"}
Nov 01 09:30:18.036017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.035993    4314 scope.go:122] "RemoveContainer" containerID="7bbf8f2f686fd3fc3157bfe8107dc1007f6fcadba46cf51074431305968f35c2"
Nov 01 09:30:18.036277 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:18.036259    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k_odh-ai-gateway-infra(31989828-2bfe-49ac-a171-540ec24af1c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" podUID="31989828-2bfe-49ac-a171-540ec24af1c0"
Nov 01 09:30:18.038671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.038600    4314 generic.go:356] "Generic (PLEG): container finished" podID="4705630f-ed5a-4364-a36c-38278704ee92" containerID="e472b33baeaa2b6bcb817ce04ab58e75970df6a918ad2b729a81c82e1752e78b" exitCode=60
Nov 01 09:30:18.038788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.038671    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerDied","Data":"e472b33baeaa2b6bcb817ce04ab58e75970df6a918ad2b729a81c82e1752e78b"}
Nov 01 09:30:18.039064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.039044    4314 scope.go:122] "RemoveContainer" containerID="e472b33baeaa2b6bcb817ce04ab58e75970df6a918ad2b729a81c82e1752e78b"
Nov 01 09:30:18.039319 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:18.039294    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72_odh-ai-gateway-infra(4705630f-ed5a-4364-a36c-38278704ee92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" podUID="4705630f-ed5a-4364-a36c-38278704ee92"
Nov 01 09:30:18.044305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.044277    4314 generic.go:356] "Generic (PLEG): container finished" podID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerID="e49de50d14a110551417136544dddd6ab948a1d7aa042cee6cc15c2aeb2c94d9" exitCode=60
Nov 01 09:30:18.044404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.044342    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerDied","Data":"e49de50d14a110551417136544dddd6ab948a1d7aa042cee6cc15c2aeb2c94d9"}
Nov 01 09:30:18.044767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.044750    4314 scope.go:122] "RemoveContainer" containerID="e49de50d14a110551417136544dddd6ab948a1d7aa042cee6cc15c2aeb2c94d9"
Nov 01 09:30:18.045038 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:18.045018    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp_odh-ai-gateway-infra(b76c0b14-cede-4b3a-80e0-cdc8beaf9724)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724"
Nov 01 09:30:18.045877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.045859    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"
Nov 01 09:30:18.045877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.045868    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5" event={"ID":"8849cd98-018b-4ff4-a9a0-02cf3ce8d86a","Type":"ContainerDied","Data":"3f50004ab7e5cc9af0ffd0886fffceda687e5da6acd8082447d3060a9cbdf539"}
Nov 01 09:30:18.053934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.053896    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk" event={"ID":"8aa0d36e-7de3-4603-8604-ee0f06ddadf1","Type":"ContainerDied","Data":"f5fb4de9da34c1f9870ee962e5ef0e3a162f2e46d3ec1d55069fef5c66f6b673"}
Nov 01 09:30:18.054068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.053988    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"
Nov 01 09:30:18.097113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.097091    4314 scope.go:122] "RemoveContainer" containerID="e70a08aee66ef3747ca067e8a81354ffb351032c601df040594d0b7ce7a4d441"
Nov 01 09:30:18.111674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.111584    4314 scope.go:122] "RemoveContainer" containerID="796566faf9c1ec551671f0b1308d80908862490c2015e14b9c358f5d7177b4a1"
Nov 01 09:30:18.126166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.126136    4314 scope.go:122] "RemoveContainer" containerID="445b097b3a585d520ec834698e5c50f472541a5a3edaafabafea7ed6576b5f8e"
Nov 01 09:30:18.139028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.139006    4314 scope.go:122] "RemoveContainer" containerID="2a3685653267f08698237d8364632bfb54fd3af1430f4df8094dc7c93de96710"
Nov 01 09:30:18.142814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.142794    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"]
Nov 01 09:30:18.151816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.151799    4314 scope.go:122] "RemoveContainer" containerID="b6df1604ecdcb74a2008079fc93c4ff2e7195003cc5b3071c53beda6ac2a0bb2"
Nov 01 09:30:18.250962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.250933    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"]
Nov 01 09:30:18.322801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.322735    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:18.433770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.433750    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:18.495247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.495215    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2\" (UniqueName: \"kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2\") pod \"aad05b42-95fc-4704-861d-ce903c61cff9\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") "
Nov 01 09:30:18.495481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.495460    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca\") pod \"aad05b42-95fc-4704-861d-ce903c61cff9\" (UID: \"aad05b42-95fc-4704-861d-ce903c61cff9\") "
Nov 01 09:30:18.495828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.495804    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca" pod "aad05b42-95fc-4704-861d-ce903c61cff9" (UID: "aad05b42-95fc-4704-861d-ce903c61cff9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:18.495974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.495956    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad05b42-95fc-4704-861d-ce903c61cff9-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.497406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.497384    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2" pod "aad05b42-95fc-4704-861d-ce903c61cff9" (UID: "aad05b42-95fc-4704-861d-ce903c61cff9"). InnerVolumeSpecName "kube-api-access-wqmn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:18.597294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.597219    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca\") pod \"5294f372-7697-4725-9642-3e189f9244a4\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") "
Nov 01 09:30:18.597434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.597334    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8\" (UniqueName: \"kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8\") pod \"5294f372-7697-4725-9642-3e189f9244a4\" (UID: \"5294f372-7697-4725-9642-3e189f9244a4\") "
Nov 01 09:30:18.597576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.597552    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca" pod "5294f372-7697-4725-9642-3e189f9244a4" (UID: "5294f372-7697-4725-9642-3e189f9244a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:18.597834 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.597815    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294f372-7697-4725-9642-3e189f9244a4-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.597903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.597838    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqmn2\" (UniqueName: \"kubernetes.io/projected/aad05b42-95fc-4704-861d-ce903c61cff9-kube-api-access-wqmn2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.599290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.599262    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8" pod "5294f372-7697-4725-9642-3e189f9244a4" (UID: "5294f372-7697-4725-9642-3e189f9244a4"). InnerVolumeSpecName "kube-api-access-gshr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:18.621326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.621286    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"]
Nov 01 09:30:18.624955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.624934    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892090-wvcf5"]
Nov 01 09:30:18.699266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.699239    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshr8\" (UniqueName: \"kubernetes.io/projected/5294f372-7697-4725-9642-3e189f9244a4-kube-api-access-gshr8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.921468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.921378    4314 scope.go:122] "RemoveContainer" containerID="1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a"
Nov 01 09:30:18.931014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.930989    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" path="/var/lib/kubelet/pods/8849cd98-018b-4ff4-a9a0-02cf3ce8d86a/volumes"
Nov 01 09:30:18.943702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.943680    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"]
Nov 01 09:30:18.947341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:18.947320    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892090-fstxk"]
Nov 01 09:30:19.047215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.047184    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"]
Nov 01 09:30:19.052756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.052730    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892090-kg4rh"]
Nov 01 09:30:19.064235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.064215    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"
Nov 01 09:30:19.064374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.064252    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd" event={"ID":"aad05b42-95fc-4704-861d-ce903c61cff9","Type":"ContainerDied","Data":"56c4b5e67c48a3e339ad86d5058bf563524d1a9461995873976700895d70f616"}
Nov 01 09:30:19.064374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.064319    4314 scope.go:122] "RemoveContainer" containerID="3eec99721827f4aeff02a93441264fc3bbfe1c3a48b4656b8ad7a48a1142facc"
Nov 01 09:30:19.065822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.065796    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"
Nov 01 09:30:19.065943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.065815    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6" event={"ID":"5294f372-7697-4725-9642-3e189f9244a4","Type":"ContainerDied","Data":"1e2abe6708a0ec5cfc74bf27079f22f8b68cf59c098a947a5e2d052a6adcd2dc"}
Nov 01 09:30:19.075350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.075329    4314 scope.go:122] "RemoveContainer" containerID="d5d778b2a1126c5015d14c733c52814fd5d47c184d2e39d59d8ec4c8b7302e5a"
Nov 01 09:30:19.218127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.218099    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"]
Nov 01 09:30:19.221594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.221571    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892090-c8lm6"]
Nov 01 09:30:19.249739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.249714    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"]
Nov 01 09:30:19.326102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.326073    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"]
Nov 01 09:30:19.337197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.337173    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892090-fgxjd"]
Nov 01 09:30:19.361203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.361175    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"]
Nov 01 09:30:19.397802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.397778    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:19.508855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.508822    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5\" (UniqueName: \"kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5\") pod \"4705630f-ed5a-4364-a36c-38278704ee92\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") "
Nov 01 09:30:19.508994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.508869    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca\") pod \"4705630f-ed5a-4364-a36c-38278704ee92\" (UID: \"4705630f-ed5a-4364-a36c-38278704ee92\") "
Nov 01 09:30:19.509218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.509196    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca" pod "4705630f-ed5a-4364-a36c-38278704ee92" (UID: "4705630f-ed5a-4364-a36c-38278704ee92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:19.510857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.510833    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5" pod "4705630f-ed5a-4364-a36c-38278704ee92" (UID: "4705630f-ed5a-4364-a36c-38278704ee92"). InnerVolumeSpecName "kube-api-access-c9bs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:19.522561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.522526    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"]
Nov 01 09:30:19.526693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.526671    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:19.611219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.611185    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bs5\" (UniqueName: \"kubernetes.io/projected/4705630f-ed5a-4364-a36c-38278704ee92-kube-api-access-c9bs5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:19.611219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.611221    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4705630f-ed5a-4364-a36c-38278704ee92-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:19.658165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.658135    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:19.712051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.712019    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca\") pod \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") "
Nov 01 09:30:19.712187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.712082    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh\" (UniqueName: \"kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh\") pod \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\" (UID: \"b76c0b14-cede-4b3a-80e0-cdc8beaf9724\") "
Nov 01 09:30:19.712356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.712336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca" pod "b76c0b14-cede-4b3a-80e0-cdc8beaf9724" (UID: "b76c0b14-cede-4b3a-80e0-cdc8beaf9724"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:19.712517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.712498    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:19.714114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.714095    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh" pod "b76c0b14-cede-4b3a-80e0-cdc8beaf9724" (UID: "b76c0b14-cede-4b3a-80e0-cdc8beaf9724"). InnerVolumeSpecName "kube-api-access-drkdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:19.812979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.812901    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca\") pod \"31989828-2bfe-49ac-a171-540ec24af1c0\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") "
Nov 01 09:30:19.813150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.813045    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2\" (UniqueName: \"kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2\") pod \"31989828-2bfe-49ac-a171-540ec24af1c0\" (UID: \"31989828-2bfe-49ac-a171-540ec24af1c0\") "
Nov 01 09:30:19.813312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.813283    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca" pod "31989828-2bfe-49ac-a171-540ec24af1c0" (UID: "31989828-2bfe-49ac-a171-540ec24af1c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:19.813615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.813591    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drkdh\" (UniqueName: \"kubernetes.io/projected/b76c0b14-cede-4b3a-80e0-cdc8beaf9724-kube-api-access-drkdh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:19.813615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.813616    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31989828-2bfe-49ac-a171-540ec24af1c0-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:19.815039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.815023    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2" pod "31989828-2bfe-49ac-a171-540ec24af1c0" (UID: "31989828-2bfe-49ac-a171-540ec24af1c0"). InnerVolumeSpecName "kube-api-access-tpwq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:19.872433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.872404    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:19.914584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:19.914556    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpwq2\" (UniqueName: \"kubernetes.io/projected/31989828-2bfe-49ac-a171-540ec24af1c0-kube-api-access-tpwq2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:20.085097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.085016    4314 generic.go:356] "Generic (PLEG): container finished" podID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerID="49c09aa76671a14cebc37d71d5e405c3711ae5ddfaf3af008f645e0f08eb572b" exitCode=60
Nov 01 09:30:20.085634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.085089    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerDied","Data":"49c09aa76671a14cebc37d71d5e405c3711ae5ddfaf3af008f645e0f08eb572b"}
Nov 01 09:30:20.085634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.085152    4314 scope.go:122] "RemoveContainer" containerID="1339ce9120994104dcc6c2f6f5ff04b1f0f14b01d36cb0fcf4a957493bfc435a"
Nov 01 09:30:20.085634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.085528    4314 scope.go:122] "RemoveContainer" containerID="49c09aa76671a14cebc37d71d5e405c3711ae5ddfaf3af008f645e0f08eb572b"
Nov 01 09:30:20.085961 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:20.085939    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff_odh-ai-gateway-infra(e434ec4a-df75-4ae0-89bc-a20ff6c3d726)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726"
Nov 01 09:30:20.086454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.086420    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"
Nov 01 09:30:20.086526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.086451    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k" event={"ID":"31989828-2bfe-49ac-a171-540ec24af1c0","Type":"ContainerDied","Data":"d585a2e68fd9dbec4afbc15d47a2b22d1ef524f9bcbaca7c219c783cc07a31ea"}
Nov 01 09:30:20.087897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.087880    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"
Nov 01 09:30:20.087897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.087889    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72" event={"ID":"4705630f-ed5a-4364-a36c-38278704ee92","Type":"ContainerDied","Data":"b71f973321413488c809b596c26db83ac300d1b9791bfaf8e94076aecfaef256"}
Nov 01 09:30:20.089250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.089227    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp" event={"ID":"b76c0b14-cede-4b3a-80e0-cdc8beaf9724","Type":"ContainerDied","Data":"b4c746029537efa848f23a9dc6c0be71e6ce4005b9b8137e6d9e19440431516e"}
Nov 01 09:30:20.089250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.089241    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"
Nov 01 09:30:20.099863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.099844    4314 scope.go:122] "RemoveContainer" containerID="7bbf8f2f686fd3fc3157bfe8107dc1007f6fcadba46cf51074431305968f35c2"
Nov 01 09:30:20.109749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.109733    4314 scope.go:122] "RemoveContainer" containerID="e472b33baeaa2b6bcb817ce04ab58e75970df6a918ad2b729a81c82e1752e78b"
Nov 01 09:30:20.121249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.121227    4314 scope.go:122] "RemoveContainer" containerID="e49de50d14a110551417136544dddd6ab948a1d7aa042cee6cc15c2aeb2c94d9"
Nov 01 09:30:20.302475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.302441    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"]
Nov 01 09:30:20.306690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.306663    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892090-flq72"]
Nov 01 09:30:20.407805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.407734    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"]
Nov 01 09:30:20.415867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.415827    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892090-sblxp"]
Nov 01 09:30:20.515246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.515214    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"]
Nov 01 09:30:20.519930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.519904    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892090-mx57k"]
Nov 01 09:30:20.662593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.662471    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:20.929524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.929451    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" path="/var/lib/kubelet/pods/31989828-2bfe-49ac-a171-540ec24af1c0/volumes"
Nov 01 09:30:20.929939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.929924    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4705630f-ed5a-4364-a36c-38278704ee92" path="/var/lib/kubelet/pods/4705630f-ed5a-4364-a36c-38278704ee92/volumes"
Nov 01 09:30:20.930291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.930280    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5294f372-7697-4725-9642-3e189f9244a4" path="/var/lib/kubelet/pods/5294f372-7697-4725-9642-3e189f9244a4/volumes"
Nov 01 09:30:20.930979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.930966    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" path="/var/lib/kubelet/pods/7b74d268-7fef-47fc-a995-fec1203c7e11/volumes"
Nov 01 09:30:20.931387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.931374    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" path="/var/lib/kubelet/pods/8aa0d36e-7de3-4603-8604-ee0f06ddadf1/volumes"
Nov 01 09:30:20.931768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.931756    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" path="/var/lib/kubelet/pods/aad05b42-95fc-4704-861d-ce903c61cff9/volumes"
Nov 01 09:30:20.932404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:20.932390    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" path="/var/lib/kubelet/pods/b76c0b14-cede-4b3a-80e0-cdc8beaf9724/volumes"
Nov 01 09:30:21.199546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.199451    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"]
Nov 01 09:30:21.200352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200330    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:21.200402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200355    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:21.200402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200375    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:21.200402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200383    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:21.200402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200396    4314 cpu_manager.go:414] "containerMap: removing container" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200402    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200412    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200418    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200432    4314 cpu_manager.go:414] "containerMap: removing container" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200439    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200449    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200455    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200464    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200471    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200498    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200506    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200525    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200548    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.200558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200560    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200567    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200600    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200608    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200621    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200630    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200639    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200646    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200675    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200681    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200693    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200701    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200711    4314 cpu_manager.go:414] "containerMap: removing container" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200717    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200730    4314 cpu_manager.go:414] "containerMap: removing container" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200737    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200753    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" containerName="test-curl-7415-10423f"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200760    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" containerName="test-curl-7415-10423f"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200774    4314 cpu_manager.go:414] "containerMap: removing container" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200781    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200936    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200952    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200963    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200973    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200981    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200991    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:21.200992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.200999    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201009    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201018    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec35e67-f606-43e3-8936-8c1df6919b47" containerName="test-curl-7415-10423f"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201028    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201037    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201045    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201053    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201061    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201074    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201082    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201091    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201100    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.201793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.201109    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8849cd98-018b-4ff4-a9a0-02cf3ce8d86a" containerName="cleanup"
Nov 01 09:30:21.206728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.206706    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"]
Nov 01 09:30:21.206941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.206923    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.256200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.256005    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-49e170-openshift-default-dockercfg-7skw9\""
Nov 01 09:30:21.267580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.261336    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"]
Nov 01 09:30:21.288902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.288823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:21.332717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.332686    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkpnd\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.332884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.332738    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.332884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.332825    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.332987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.332917    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.332987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.332971    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.333079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.333007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.333151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.333124    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.333250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.333234    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.333337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.333318    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434342    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434421    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434606    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkpnd\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434645    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434670    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434724    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434798    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434834    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434839    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.434933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.434866    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.435105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.435071    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.435661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.435635    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.435904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.435862    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.435904 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.435888    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.437355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.437336    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.437625 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.437607    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.464825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.464773    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:21.493136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.493110    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkpnd\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.493290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.493117    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token\") pod \"e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") " pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.522667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.522646    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:21.544223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.544192    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:21.636601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.636567    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca\") pod \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") "
Nov 01 09:30:21.636771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.636757    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj\" (UniqueName: \"kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj\") pod \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\" (UID: \"e434ec4a-df75-4ae0-89bc-a20ff6c3d726\") "
Nov 01 09:30:21.637041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.637012    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca" pod "e434ec4a-df75-4ae0-89bc-a20ff6c3d726" (UID: "e434ec4a-df75-4ae0-89bc-a20ff6c3d726"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:21.637430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.637409    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:21.639619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.639593    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj" pod "e434ec4a-df75-4ae0-89bc-a20ff6c3d726" (UID: "e434ec4a-df75-4ae0-89bc-a20ff6c3d726"). InnerVolumeSpecName "kube-api-access-lp9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:21.738702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.738674    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9qj\" (UniqueName: \"kubernetes.io/projected/e434ec4a-df75-4ae0-89bc-a20ff6c3d726-kube-api-access-lp9qj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:21.959102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.959076    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"]
Nov 01 09:30:21.959760 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:21.959730    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2fd9c0_fdbd_4936_9642_da1174059da4.slice/crio-9c4d194ad3767089fb1954aeac77033bd1778212a4fe1e690e00af93e6c97ffe WatchSource:0}: Error finding container 9c4d194ad3767089fb1954aeac77033bd1778212a4fe1e690e00af93e6c97ffe: Status 404 returned error can't find the container with id 9c4d194ad3767089fb1954aeac77033bd1778212a4fe1e690e00af93e6c97ffe
Nov 01 09:30:21.962546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.962497    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:21.962640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.962604    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:21.962684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:21.962650    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:22.113305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.113273    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"
Nov 01 09:30:22.113507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.113297    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff" event={"ID":"e434ec4a-df75-4ae0-89bc-a20ff6c3d726","Type":"ContainerDied","Data":"ad0bd9d12dad9ae24869b929a12510dda4aa40224d18d9655e4696c954ff3cfd"}
Nov 01 09:30:22.113507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.113367    4314 scope.go:122] "RemoveContainer" containerID="49c09aa76671a14cebc37d71d5e405c3711ae5ddfaf3af008f645e0f08eb572b"
Nov 01 09:30:22.115701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.115673    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" event={"ID":"4a2fd9c0-fdbd-4936-9642-da1174059da4","Type":"ContainerStarted","Data":"3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"}
Nov 01 09:30:22.115825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.115722    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" event={"ID":"4a2fd9c0-fdbd-4936-9642-da1174059da4","Type":"ContainerStarted","Data":"9c4d194ad3767089fb1954aeac77033bd1778212a4fe1e690e00af93e6c97ffe"}
Nov 01 09:30:22.278026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.277921    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podStartSLOduration=1.277901667 podStartE2EDuration="1.277901667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:22.275668476 +0000 UTC m=+3495.470063054" watchObservedRunningTime="2026-11-01 09:30:22.277901667 +0000 UTC m=+3495.472296221"
Nov 01 09:30:22.381801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.381762    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"]
Nov 01 09:30:22.387381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.387352    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892090-vj2ff"]
Nov 01 09:30:22.523698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.523665    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:22.873967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.873942    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:22.933717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:22.933671    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" path="/var/lib/kubelet/pods/e434ec4a-df75-4ae0-89bc-a20ff6c3d726/volumes"
Nov 01 09:30:23.530741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:23.530713    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:24.130470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:24.130439    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:24.131250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:24.131233    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:30:25.033382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:25.033351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:25.310939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:25.310865    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:25.491737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:25.491708    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:28.234328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.234301    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:28.404053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.404014    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:28.556342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.556254    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:28.623585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.623551    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"]
Nov 01 09:30:28.624230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624210    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.624230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624231    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624246    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624253    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4705630f-ed5a-4364-a36c-38278704ee92" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624262    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624269    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624301    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624307    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624351    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624360    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624376    4314 cpu_manager.go:414] "containerMap: removing container" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624383    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624393    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624400    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624409    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:28.624412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624416    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624580    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624598    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624609    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b74d268-7fef-47fc-a995-fec1203c7e11" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624620    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="31989828-2bfe-49ac-a171-540ec24af1c0" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624631    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5294f372-7697-4725-9642-3e189f9244a4" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624640    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad05b42-95fc-4704-861d-ce903c61cff9" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624654    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76c0b14-cede-4b3a-80e0-cdc8beaf9724" containerName="cleanup"
Nov 01 09:30:28.625135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.624663    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa0d36e-7de3-4603-8604-ee0f06ddadf1" containerName="cleanup"
Nov 01 09:30:28.630205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.630182    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.692695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.690873    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-49e170-dockercfg-ws6bx\""
Nov 01 09:30:28.738555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.737821    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-49e170\""
Nov 01 09:30:28.746344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.743259    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.746344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.746089    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxx2\" (UniqueName: \"kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2\") pod \"payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.746344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.745711    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"]
Nov 01 09:30:28.820094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.819996    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"]
Nov 01 09:30:28.820688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.820667    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.820688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.820685    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.820875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.820808    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e434ec4a-df75-4ae0-89bc-a20ff6c3d726" containerName="cleanup"
Nov 01 09:30:28.824808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.824782    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:28.852890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.850734    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgm6\" (UniqueName: \"kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6\") pod \"payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:28.852890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.850837    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxx2\" (UniqueName: \"kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2\") pod \"payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.852890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.850885    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.852890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.850905    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:28.852890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.852371    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.898632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.898600    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"]
Nov 01 09:30:28.942981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.942948    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxx2\" (UniqueName: \"kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2\") pod \"payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.946733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.946709    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:28.952681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.952658    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:28.952854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.952836    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgm6\" (UniqueName: \"kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6\") pod \"payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:28.953398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:28.953374    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:29.010061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:29.010013    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgm6\" (UniqueName: \"kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6\") pod \"payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:29.145728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:29.145639    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:29.348277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:29.348249    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"]
Nov 01 09:30:29.349140 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:29.349113    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc3a2ec_8f92_4ac4_ad2c_3134008a0ea7.slice/crio-0d93d4c7da737033708e5a9ad56c7ab8d16c666c59258bade97266ccedbb2af9 WatchSource:0}: Error finding container 0d93d4c7da737033708e5a9ad56c7ab8d16c666c59258bade97266ccedbb2af9: Status 404 returned error can't find the container with id 0d93d4c7da737033708e5a9ad56c7ab8d16c666c59258bade97266ccedbb2af9
Nov 01 09:30:29.546447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:29.546420    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"]
Nov 01 09:30:29.547406 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:29.547379    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c3635b_2510_4128_808a_3b03c5a34e9a.slice/crio-9475805188d5a12950271002833533aa2f89e61fe3ded730e54268e631dcc32c WatchSource:0}: Error finding container 9475805188d5a12950271002833533aa2f89e61fe3ded730e54268e631dcc32c: Status 404 returned error can't find the container with id 9475805188d5a12950271002833533aa2f89e61fe3ded730e54268e631dcc32c
Nov 01 09:30:30.199893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.199848    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh" event={"ID":"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7","Type":"ContainerStarted","Data":"e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26"}
Nov 01 09:30:30.200109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.199918    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh" event={"ID":"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7","Type":"ContainerStarted","Data":"0d93d4c7da737033708e5a9ad56c7ab8d16c666c59258bade97266ccedbb2af9"}
Nov 01 09:30:30.200109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.199944    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:30.201833 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.201803    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5" event={"ID":"40c3635b-2510-4128-808a-3b03c5a34e9a","Type":"ContainerStarted","Data":"e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584"}
Nov 01 09:30:30.201985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.201853    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5" event={"ID":"40c3635b-2510-4128-808a-3b03c5a34e9a","Type":"ContainerStarted","Data":"9475805188d5a12950271002833533aa2f89e61fe3ded730e54268e631dcc32c"}
Nov 01 09:30:30.201985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.201884    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:30.309416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.309354    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh" podStartSLOduration=2.309335658 podStartE2EDuration="2.309335658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:30.30738409 +0000 UTC m=+3503.501778650" watchObservedRunningTime="2026-11-01 09:30:30.309335658 +0000 UTC m=+3503.503730214"
Nov 01 09:30:30.422192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:30.422124    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5" podStartSLOduration=2.422105354 podStartE2EDuration="2.422105354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:30.417193156 +0000 UTC m=+3503.611587713" watchObservedRunningTime="2026-11-01 09:30:30.422105354 +0000 UTC m=+3503.616499910"
Nov 01 09:30:32.437566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:32.437501    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:33.176468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:33.176341    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:36.246708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:36.246678    4314 generic.go:356] "Generic (PLEG): container finished" podID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerID="c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964" exitCode=0
Nov 01 09:30:36.247120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:36.246752    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" event={"ID":"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06","Type":"ContainerDied","Data":"c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"}
Nov 01 09:30:36.247171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:36.247154    4314 scope.go:122] "RemoveContainer" containerID="c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"
Nov 01 09:30:36.248823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:36.248791    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:36.248924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:36.248860    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:36.248924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:36.248893    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:37.255390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:37.255351    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" event={"ID":"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06","Type":"ContainerStarted","Data":"95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623"}
Nov 01 09:30:37.354695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:37.354661    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:30:37.355768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:37.355746    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 01 09:30:37.355858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:37.355793    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 01 09:30:38.001446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:38.001412    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:38.355210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:38.355137    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 01 09:30:38.355210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:38.355185    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 01 09:30:39.354813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:39.354776    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 01 09:30:39.354992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:39.354842    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 01 09:30:40.354868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:40.354832    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 01 09:30:40.355277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:40.354891    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 01 09:30:41.150495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:41.150454    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:30:41.211665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:41.211631    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:30:41.211846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:41.211719    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:30:41.323225 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:30:41.323188    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:30:41.354932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:41.354893    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 01 09:30:41.355407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:41.354949    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 01 09:30:42.411964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:42.411843    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:30:42.413348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:42.413327    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:30:42.779169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:42.779135    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:46.319305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:46.319275    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:46.908570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:46.908514    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"]
Nov 01 09:30:46.938521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:46.938492    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:46.975849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:46.975818    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:46.985960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:46.985936    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"]
Nov 01 09:30:47.027829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.027796    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-25d9fe-openshift-default-dockercfg-mxzcx\""
Nov 01 09:30:47.117980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.117891    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.117980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.117949    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.117986    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.118024    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bz74\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.118091    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.118177    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.118200    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.118367    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.118511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.118401    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.126442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.125615    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:47.220329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220228    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220265    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220351    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220370    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220388    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220412    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bz74\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220447    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220499    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.220626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220524    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.221035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220738    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.221035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.220981    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.221148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.221055    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.221148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.221080    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.221230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.221188    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.222752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.222731    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.223044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.223024    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.277733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.277706    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.278092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.278068    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bz74\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74\") pod \"e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") " pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.563953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.563909    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:47.715613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.715523    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:47.988344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:47.988310    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"]
Nov 01 09:30:48.020867 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:48.020840    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fe3e44_434b_47d5_88e6_de80b6d2e0cc.slice/crio-41549e1a31f032b95fb7e32926419f3e66e51c87eb5fb1363faaa2843f16dcd1 WatchSource:0}: Error finding container 41549e1a31f032b95fb7e32926419f3e66e51c87eb5fb1363faaa2843f16dcd1: Status 404 returned error can't find the container with id 41549e1a31f032b95fb7e32926419f3e66e51c87eb5fb1363faaa2843f16dcd1
Nov 01 09:30:48.023564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.023514    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:48.023659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.023602    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:48.023659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.023636    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:30:48.361287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.361202    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" event={"ID":"22fe3e44-434b-47d5-88e6-de80b6d2e0cc","Type":"ContainerStarted","Data":"bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"}
Nov 01 09:30:48.361287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.361256    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" event={"ID":"22fe3e44-434b-47d5-88e6-de80b6d2e0cc","Type":"ContainerStarted","Data":"41549e1a31f032b95fb7e32926419f3e66e51c87eb5fb1363faaa2843f16dcd1"}
Nov 01 09:30:48.476555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.476477    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podStartSLOduration=2.476463577 podStartE2EDuration="2.476463577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:48.473925382 +0000 UTC m=+3521.668319944" watchObservedRunningTime="2026-11-01 09:30:48.476463577 +0000 UTC m=+3521.670858130"
Nov 01 09:30:48.564455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:48.564423    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:49.570405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:49.570373    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:50.374977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:50.374944    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:50.375757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:50.375732    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:30:51.252577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:51.252521    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:51.510038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:51.509953    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:51.658681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:51.658645    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:52.870377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:52.870346    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:54.423162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.423132    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:54.582653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.582618    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:30:54.825065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.825031    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"]
Nov 01 09:30:54.829516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.829491    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:54.915801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.915764    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:54.916010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.915816    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9q9k\" (UniqueName: \"kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k\") pod \"payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:54.920675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.920646    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-25d9fe\""
Nov 01 09:30:54.924035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.924007    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-25d9fe-dockercfg-tmr8n\""
Nov 01 09:30:54.933598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.933569    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"]
Nov 01 09:30:54.992590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.992554    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"]
Nov 01 09:30:54.998182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:54.998158    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.016963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.016928    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:55.017120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.016977    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q9k\" (UniqueName: \"kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k\") pod \"payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:55.017120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.017012    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnjh\" (UniqueName: \"kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh\") pod \"payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.017120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.017086    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.017860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.017838    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:55.054050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.054023    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"]
Nov 01 09:30:55.074251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.074219    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q9k\" (UniqueName: \"kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k\") pod \"payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:55.118025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.117951    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnjh\" (UniqueName: \"kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh\") pod \"payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.118162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.118051    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.118744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.118727    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.146174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.146148    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:55.175183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.175149    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnjh\" (UniqueName: \"kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh\") pod \"payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.316228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.316196    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:55.517286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.517253    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"]
Nov 01 09:30:55.517706 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:55.517521    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c67b251_a132_46ff_82eb_11985fe5814f.slice/crio-cf9d51fa809da1121f88d223ffceb82f9de2a427e91a14207cd631555865d740 WatchSource:0}: Error finding container cf9d51fa809da1121f88d223ffceb82f9de2a427e91a14207cd631555865d740: Status 404 returned error can't find the container with id cf9d51fa809da1121f88d223ffceb82f9de2a427e91a14207cd631555865d740
Nov 01 09:30:55.687598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:55.687566    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"]
Nov 01 09:30:55.688326 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:30:55.688292    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc493ba1b_ba21_400a_b40a_9bea6cc5f7fc.slice/crio-ed1e9d599e06595e5a6860efe9e0653273945c5c6178ec390080d87f12b22d15 WatchSource:0}: Error finding container ed1e9d599e06595e5a6860efe9e0653273945c5c6178ec390080d87f12b22d15: Status 404 returned error can't find the container with id ed1e9d599e06595e5a6860efe9e0653273945c5c6178ec390080d87f12b22d15
Nov 01 09:30:56.422727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.422686    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp" event={"ID":"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc","Type":"ContainerStarted","Data":"76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05"}
Nov 01 09:30:56.422938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.422746    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp" event={"ID":"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc","Type":"ContainerStarted","Data":"ed1e9d599e06595e5a6860efe9e0653273945c5c6178ec390080d87f12b22d15"}
Nov 01 09:30:56.422938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.422775    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:30:56.424330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.424296    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2" event={"ID":"8c67b251-a132-46ff-82eb-11985fe5814f","Type":"ContainerStarted","Data":"05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc"}
Nov 01 09:30:56.424464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.424351    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2" event={"ID":"8c67b251-a132-46ff-82eb-11985fe5814f","Type":"ContainerStarted","Data":"cf9d51fa809da1121f88d223ffceb82f9de2a427e91a14207cd631555865d740"}
Nov 01 09:30:56.424464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.424408    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:30:56.532806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.532747    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp" podStartSLOduration=2.532732751 podStartE2EDuration="2.532732751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:56.530238639 +0000 UTC m=+3529.724633205" watchObservedRunningTime="2026-11-01 09:30:56.532732751 +0000 UTC m=+3529.727127330"
Nov 01 09:30:56.643393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:56.643341    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2" podStartSLOduration=2.643328277 podStartE2EDuration="2.643328277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:56.63799161 +0000 UTC m=+3529.832386193" watchObservedRunningTime="2026-11-01 09:30:56.643328277 +0000 UTC m=+3529.837722831"
Nov 01 09:30:58.064875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:30:58.064841    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:02.844803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:02.844759    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:04.449829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:04.449794    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:05.064461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:05.064431    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:05.270661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:05.263604    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:05.383006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:05.382935    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"]
Nov 01 09:31:05.413701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:05.413657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:07.436296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:07.436265    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:31:07.436805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:07.436354    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:31:07.690698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:07.690619    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:08.148178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:08.148140    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"]
Nov 01 09:31:08.185680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:08.185653    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:09.759992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:09.759958    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:09.940838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:09.940800    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:10.139720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:10.139640    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:11.330619 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:31:11.330581    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:31:12.876774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:12.876740    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:13.022278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:13.022243    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:13.166835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:13.166747    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:17.880278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:17.880239    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:20.259594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.259491    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:20.403379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.403336    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5" podUID="5e9d6bba-be30-4171-a6e9-56bf01d597dd" containerName="main" containerID="cri-o://74566c541c360491d05ff4acb226bc5207b4a153004550fea1e965a516ceadde" gracePeriod=45
Nov 01 09:31:20.600832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.600802    4314 generic.go:356] "Generic (PLEG): container finished" podID="5e9d6bba-be30-4171-a6e9-56bf01d597dd" containerID="74566c541c360491d05ff4acb226bc5207b4a153004550fea1e965a516ceadde" exitCode=0
Nov 01 09:31:20.600989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.600880    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5" event={"ID":"5e9d6bba-be30-4171-a6e9-56bf01d597dd","Type":"ContainerDied","Data":"74566c541c360491d05ff4acb226bc5207b4a153004550fea1e965a516ceadde"}
Nov 01 09:31:20.671781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.671754    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:31:20.786113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786020    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm\") pod \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") "
Nov 01 09:31:20.786301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786147    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home\") pod \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") "
Nov 01 09:31:20.786301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786190    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x\" (UniqueName: \"kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x\") pod \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") "
Nov 01 09:31:20.786301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786295    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache\") pod \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") "
Nov 01 09:31:20.786486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786372    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir\") pod \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") "
Nov 01 09:31:20.786486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786452    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs\") pod \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\" (UID: \"5e9d6bba-be30-4171-a6e9-56bf01d597dd\") "
Nov 01 09:31:20.786916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786837    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache" pod "5e9d6bba-be30-4171-a6e9-56bf01d597dd" (UID: "5e9d6bba-be30-4171-a6e9-56bf01d597dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:20.787024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.786962    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home" pod "5e9d6bba-be30-4171-a6e9-56bf01d597dd" (UID: "5e9d6bba-be30-4171-a6e9-56bf01d597dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:20.787185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.787143    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir" pod "5e9d6bba-be30-4171-a6e9-56bf01d597dd" (UID: "5e9d6bba-be30-4171-a6e9-56bf01d597dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:20.787311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.787292    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:20.787380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.787316    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:20.787380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.787330    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:20.789331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.789301    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm" pod "5e9d6bba-be30-4171-a6e9-56bf01d597dd" (UID: "5e9d6bba-be30-4171-a6e9-56bf01d597dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:20.789592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.789520    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs" pod "5e9d6bba-be30-4171-a6e9-56bf01d597dd" (UID: "5e9d6bba-be30-4171-a6e9-56bf01d597dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:31:20.791063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.791035    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x" pod "5e9d6bba-be30-4171-a6e9-56bf01d597dd" (UID: "5e9d6bba-be30-4171-a6e9-56bf01d597dd"). InnerVolumeSpecName "kube-api-access-h8q7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:31:20.892182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.889561    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9d6bba-be30-4171-a6e9-56bf01d597dd-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:20.892182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.889610    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9d6bba-be30-4171-a6e9-56bf01d597dd-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:20.892182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:20.889639    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8q7x\" (UniqueName: \"kubernetes.io/projected/5e9d6bba-be30-4171-a6e9-56bf01d597dd-kube-api-access-h8q7x\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:21.609935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:21.609902    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"
Nov 01 09:31:21.609935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:21.609915    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5" event={"ID":"5e9d6bba-be30-4171-a6e9-56bf01d597dd","Type":"ContainerDied","Data":"b8a849db7806d5426c24856aee61a339324cc28ef87c0b8d6313013e28021d5a"}
Nov 01 09:31:21.610517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:21.609970    4314 scope.go:122] "RemoveContainer" containerID="74566c541c360491d05ff4acb226bc5207b4a153004550fea1e965a516ceadde"
Nov 01 09:31:21.766888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:21.766850    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"]
Nov 01 09:31:21.813315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:21.813277    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/auth-test-model-65aeef-kserve-7f9964f5bf-dq7d5"]
Nov 01 09:31:22.696670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:22.696636    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:22.929480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:22.929443    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9d6bba-be30-4171-a6e9-56bf01d597dd" path="/var/lib/kubelet/pods/5e9d6bba-be30-4171-a6e9-56bf01d597dd/volumes"
Nov 01 09:31:23.534441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.534405    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th" podUID="b2eb30c5-5145-4f75-8084-ae7017186ce8" containerName="main" containerID="cri-o://f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de" gracePeriod=45
Nov 01 09:31:23.805280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.805257    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:31:23.926929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.926899    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf\" (UniqueName: \"kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf\") pod \"b2eb30c5-5145-4f75-8084-ae7017186ce8\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") "
Nov 01 09:31:23.927115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.926987    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache\") pod \"b2eb30c5-5145-4f75-8084-ae7017186ce8\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") "
Nov 01 09:31:23.927115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927093    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs\") pod \"b2eb30c5-5145-4f75-8084-ae7017186ce8\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") "
Nov 01 09:31:23.927476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927140    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir\") pod \"b2eb30c5-5145-4f75-8084-ae7017186ce8\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") "
Nov 01 09:31:23.927476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927212    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm\") pod \"b2eb30c5-5145-4f75-8084-ae7017186ce8\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") "
Nov 01 09:31:23.927476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927232    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache" pod "b2eb30c5-5145-4f75-8084-ae7017186ce8" (UID: "b2eb30c5-5145-4f75-8084-ae7017186ce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:23.927476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927293    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home\") pod \"b2eb30c5-5145-4f75-8084-ae7017186ce8\" (UID: \"b2eb30c5-5145-4f75-8084-ae7017186ce8\") "
Nov 01 09:31:23.927770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927557    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home" pod "b2eb30c5-5145-4f75-8084-ae7017186ce8" (UID: "b2eb30c5-5145-4f75-8084-ae7017186ce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:23.927770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927604    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir" pod "b2eb30c5-5145-4f75-8084-ae7017186ce8" (UID: "b2eb30c5-5145-4f75-8084-ae7017186ce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:23.927960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927941    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:23.928009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927968    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:23.928009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.927984    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:23.930290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.930265    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf" pod "b2eb30c5-5145-4f75-8084-ae7017186ce8" (UID: "b2eb30c5-5145-4f75-8084-ae7017186ce8"). InnerVolumeSpecName "kube-api-access-hpptf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:31:23.930402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.930392    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs" pod "b2eb30c5-5145-4f75-8084-ae7017186ce8" (UID: "b2eb30c5-5145-4f75-8084-ae7017186ce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:31:23.930558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:23.930518    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm" pod "b2eb30c5-5145-4f75-8084-ae7017186ce8" (UID: "b2eb30c5-5145-4f75-8084-ae7017186ce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:31:24.029354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.029313    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpptf\" (UniqueName: \"kubernetes.io/projected/b2eb30c5-5145-4f75-8084-ae7017186ce8-kube-api-access-hpptf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:24.029354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.029351    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2eb30c5-5145-4f75-8084-ae7017186ce8-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:24.029613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.029368    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2eb30c5-5145-4f75-8084-ae7017186ce8-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:31:24.640942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.640904    4314 generic.go:356] "Generic (PLEG): container finished" podID="b2eb30c5-5145-4f75-8084-ae7017186ce8" containerID="f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de" exitCode=0
Nov 01 09:31:24.641099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.641073    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th" event={"ID":"b2eb30c5-5145-4f75-8084-ae7017186ce8","Type":"ContainerDied","Data":"f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de"}
Nov 01 09:31:24.641178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.641122    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th" event={"ID":"b2eb30c5-5145-4f75-8084-ae7017186ce8","Type":"ContainerDied","Data":"296c9923f25687a9be0d37198ecc96b3edd17d36536d6595abede96345824352"}
Nov 01 09:31:24.641178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.641145    4314 scope.go:122] "RemoveContainer" containerID="f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de"
Nov 01 09:31:24.641353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.641336    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"
Nov 01 09:31:24.655160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.655137    4314 scope.go:122] "RemoveContainer" containerID="f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de"
Nov 01 09:31:24.655501 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:31:24.655479    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de\": container with ID starting with f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de not found: ID does not exist" containerID="f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de"
Nov 01 09:31:24.655589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.655509    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de"} err="failed to get container status \"f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de\": rpc error: code = NotFound desc = could not find container \"f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de\": container with ID starting with f01e0c6406f83b2f8d56e47a28d37a3e903a8de678b59fac929f7cf74324e3de not found: ID does not exist"
Nov 01 09:31:24.797080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.797046    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"]
Nov 01 09:31:24.803040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.803012    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/auth-test-model-46bf99-kserve-666d56fc8f-rg7th"]
Nov 01 09:31:24.929974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:24.929895    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2eb30c5-5145-4f75-8084-ae7017186ce8" path="/var/lib/kubelet/pods/b2eb30c5-5145-4f75-8084-ae7017186ce8/volumes"
Nov 01 09:31:27.585732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:27.585699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:31.557952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:31.557917    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:31.990406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:31.990368    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:32.505480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.505444    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"]
Nov 01 09:31:32.506081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.506067    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b2eb30c5-5145-4f75-8084-ae7017186ce8" containerName="main"
Nov 01 09:31:32.506126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.506082    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2eb30c5-5145-4f75-8084-ae7017186ce8" containerName="main"
Nov 01 09:31:32.506126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.506111    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e9d6bba-be30-4171-a6e9-56bf01d597dd" containerName="main"
Nov 01 09:31:32.506126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.506116    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9d6bba-be30-4171-a6e9-56bf01d597dd" containerName="main"
Nov 01 09:31:32.506229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.506223    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2eb30c5-5145-4f75-8084-ae7017186ce8" containerName="main"
Nov 01 09:31:32.506264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.506237    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9d6bba-be30-4171-a6e9-56bf01d597dd" containerName="main"
Nov 01 09:31:32.511206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.511188    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531322    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcc7b\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531357    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531434    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531452    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531621    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531735    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531827    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.531907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.531896    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.560165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.560131    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f8071b-openshift-default-dockercfg-db52s\""
Nov 01 09:31:32.608358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.608319    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:32.611897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.611867    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"]
Nov 01 09:31:32.633482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.633445    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.633670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.633510    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.633670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.633579    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.633858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.633836    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.634041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634027    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.634489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634465    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcc7b\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.634489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634114    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.634745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634509    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.634745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634612    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.634745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634642    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.635039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634383    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.635117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.634323    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.635117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.635045    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.635382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.635282    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.636736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.636715    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.642800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.642776    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.711920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.709422    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.720944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.720901    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcc7b\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b\") pod \"e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") " pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.825657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.825571    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:32.903621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:32.903524    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:33.091638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.091547    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:33.258004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.257976    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"]
Nov 01 09:31:33.258685 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:31:33.258631    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a5bfcf_d256_4430_99b9_8c5c3d599f4c.slice/crio-80067e9b03a91d388db8d80a158fb5a39df652114b757e12567b3f7b5d5f1afd WatchSource:0}: Error finding container 80067e9b03a91d388db8d80a158fb5a39df652114b757e12567b3f7b5d5f1afd: Status 404 returned error can't find the container with id 80067e9b03a91d388db8d80a158fb5a39df652114b757e12567b3f7b5d5f1afd
Nov 01 09:31:33.261845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.261806    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:31:33.261963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.261880    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:31:33.261963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.261916    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:31:33.737360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.737323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" event={"ID":"59a5bfcf-d256-4430-99b9-8c5c3d599f4c","Type":"ContainerStarted","Data":"0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"}
Nov 01 09:31:33.737838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.737391    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" event={"ID":"59a5bfcf-d256-4430-99b9-8c5c3d599f4c","Type":"ContainerStarted","Data":"80067e9b03a91d388db8d80a158fb5a39df652114b757e12567b3f7b5d5f1afd"}
Nov 01 09:31:33.826383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.826342    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:33.852000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:33.851953    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podStartSLOduration=1.851938852 podStartE2EDuration="1.851938852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:33.849891751 +0000 UTC m=+3567.044286309" watchObservedRunningTime="2026-11-01 09:31:33.851938852 +0000 UTC m=+3567.046333407"
Nov 01 09:31:34.832406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:34.832379    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:35.751309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:35.751276    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:35.752393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:35.752376    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:31:36.652950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:36.652920    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:36.837063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:36.837025    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:36.980420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:36.980392    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:37.911271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:37.911237    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:41.322603 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:31:41.322564    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:31:41.572944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:41.572856    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:41.734949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:41.734918    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:41.977781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:41.977699    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"]
Nov 01 09:31:41.981872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:41.981854    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.071594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.071469    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.072113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.072084    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwzk\" (UniqueName: \"kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk\") pod \"payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.077690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.077665    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f8071b-dockercfg-ggrzh\""
Nov 01 09:31:42.078015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.077999    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f8071b\""
Nov 01 09:31:42.089898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.089868    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"]
Nov 01 09:31:42.148682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.148646    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"]
Nov 01 09:31:42.152795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.152778    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.173889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.173863    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.174035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.173907    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwzk\" (UniqueName: \"kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk\") pod \"payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.174522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.174503    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.216782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.216742    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"]
Nov 01 09:31:42.231655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.231627    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwzk\" (UniqueName: \"kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk\") pod \"payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.275251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.275215    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.275437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.275387    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9gr\" (UniqueName: \"kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr\") pod \"payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.295050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.295015    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.376707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.376666    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9gr\" (UniqueName: \"kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr\") pod \"payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.377177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.376898    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.377898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.377872    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.434069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.434035    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9gr\" (UniqueName: \"kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr\") pod \"payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.467937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.467896    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:42.666478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.666442    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"]
Nov 01 09:31:42.667127 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:31:42.667101    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fedd6d_c351_462f_b8b1_e3193d5fb7d9.slice/crio-1e7e353eb8b0c44d40e36e87c0e7e7c3c2b017543f185ff64fe50c663f6338ff WatchSource:0}: Error finding container 1e7e353eb8b0c44d40e36e87c0e7e7c3c2b017543f185ff64fe50c663f6338ff: Status 404 returned error can't find the container with id 1e7e353eb8b0c44d40e36e87c0e7e7c3c2b017543f185ff64fe50c663f6338ff
Nov 01 09:31:42.798604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.798566    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q" event={"ID":"70fedd6d-c351-462f-b8b1-e3193d5fb7d9","Type":"ContainerStarted","Data":"adcbd9f9120b7a2562b79f43d0677028ad05015d4e968504f3aa1e010d13b838"}
Nov 01 09:31:42.798792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.798633    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:42.798792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.798653    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q" event={"ID":"70fedd6d-c351-462f-b8b1-e3193d5fb7d9","Type":"ContainerStarted","Data":"1e7e353eb8b0c44d40e36e87c0e7e7c3c2b017543f185ff64fe50c663f6338ff"}
Nov 01 09:31:42.841011 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:31:42.840976    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceffbb2d_316d_4fff_92cc_c20fdb13ad56.slice/crio-ce34dac796ddb8bfba3d9f09680dbcdb44f8b355810544da0522662ba2d2847a WatchSource:0}: Error finding container ce34dac796ddb8bfba3d9f09680dbcdb44f8b355810544da0522662ba2d2847a: Status 404 returned error can't find the container with id ce34dac796ddb8bfba3d9f09680dbcdb44f8b355810544da0522662ba2d2847a
Nov 01 09:31:42.841705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.841680    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"]
Nov 01 09:31:42.912676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:42.912625    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q" podStartSLOduration=1.912609107 podStartE2EDuration="1.912609107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:42.907972729 +0000 UTC m=+3576.102367286" watchObservedRunningTime="2026-11-01 09:31:42.912609107 +0000 UTC m=+3576.107003697"
Nov 01 09:31:43.133540 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.133420    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:43.296365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.296329    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:43.438764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.438683    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:43.806699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.806661    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb" event={"ID":"ceffbb2d-316d-4fff-92cc-c20fdb13ad56","Type":"ContainerStarted","Data":"e205666664521652a2503209edb6ed976d785029f96219553df2b84c0147eed0"}
Nov 01 09:31:43.806896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.806727    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb" event={"ID":"ceffbb2d-316d-4fff-92cc-c20fdb13ad56","Type":"ContainerStarted","Data":"ce34dac796ddb8bfba3d9f09680dbcdb44f8b355810544da0522662ba2d2847a"}
Nov 01 09:31:43.806959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.806910    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:43.916667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:43.916609    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb" podStartSLOduration=1.916595219 podStartE2EDuration="1.916595219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:43.91437312 +0000 UTC m=+3577.108767678" watchObservedRunningTime="2026-11-01 09:31:43.916595219 +0000 UTC m=+3577.110989775"
Nov 01 09:31:48.439616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:48.439578    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:53.201926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:53.201889    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:31:53.809898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:53.809868    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:31:54.819380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:54.819345    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:31:57.969669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:31:57.969619    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:02.762033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:02.762000    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:07.561931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:07.561895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:11.327893 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:32:11.327852    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:32:12.335229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:12.335197    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:12.503129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:12.503096    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:12.684049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:12.683956    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:17.388042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:17.387985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:18.072805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:18.072775    4314 generic.go:356] "Generic (PLEG): container finished" podID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerID="3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86" exitCode=0
Nov 01 09:32:18.073035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:18.072851    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" event={"ID":"4a2fd9c0-fdbd-4936-9642-da1174059da4","Type":"ContainerDied","Data":"3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"}
Nov 01 09:32:18.073279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:18.073261    4314 scope.go:122] "RemoveContainer" containerID="3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"
Nov 01 09:32:18.074824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:18.074797    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:32:18.074902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:18.074866    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:32:18.074949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:18.074902    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:32:19.088950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:19.088907    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" event={"ID":"4a2fd9c0-fdbd-4936-9642-da1174059da4","Type":"ContainerStarted","Data":"dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7"}
Nov 01 09:32:19.523094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:19.523065    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:32:19.524080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:19.524053    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 01 09:32:19.524202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:19.524113    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 01 09:32:20.523672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:20.523639    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 01 09:32:20.524078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:20.523691    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 01 09:32:20.584051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:20.584018    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:21.523666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:21.523631    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 01 09:32:21.523861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:21.523697    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 01 09:32:22.184132 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:22.184080    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:22.523621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:22.523588    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 01 09:32:22.523803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:22.523654    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 01 09:32:23.523568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:23.523520    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 01 09:32:23.523973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:23.523601    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 01 09:32:24.130370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:24.130333    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:32:24.528144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:24.528113    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:32:25.131820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:25.131791    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:32:26.949357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:26.949297    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:31.700713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:31.700675    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:36.485526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:36.485488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:36.647343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:36.647228    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:40.257582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:40.257522    4314 generic.go:356] "Generic (PLEG): container finished" podID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerID="bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da" exitCode=0
Nov 01 09:32:40.257582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:40.257563    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" event={"ID":"22fe3e44-434b-47d5-88e6-de80b6d2e0cc","Type":"ContainerDied","Data":"bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"}
Nov 01 09:32:40.258184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:40.258035    4314 scope.go:122] "RemoveContainer" containerID="bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"
Nov 01 09:32:40.259878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:40.259849    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:32:40.259994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:40.259920    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:32:40.259994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:40.259952    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:32:41.287809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:41.287776    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" event={"ID":"22fe3e44-434b-47d5-88e6-de80b6d2e0cc","Type":"ContainerStarted","Data":"cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b"}
Nov 01 09:32:41.324755 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:32:41.324723    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:32:41.485519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:41.485486    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:41.572074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:41.571977    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:32:41.572444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:41.572420    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Nov 01 09:32:41.572666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:41.572640    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Nov 01 09:32:42.578992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:42.578952    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Nov 01 09:32:42.579645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:42.579611    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Nov 01 09:32:43.575946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:43.575907    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Nov 01 09:32:43.576263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:43.576231    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Nov 01 09:32:44.574045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:44.574002    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Nov 01 09:32:44.574700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:44.574664    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Nov 01 09:32:45.577701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:45.577659    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Nov 01 09:32:45.578299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:45.578267    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Nov 01 09:32:46.384164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:46.384117    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:46.571011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:46.570971    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Nov 01 09:32:46.571228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:46.571146    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Nov 01 09:32:47.569376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:47.569341    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:32:47.569880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:47.569747    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:32:47.570544 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:47.570512    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:32:51.186714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:51.186632    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:32:55.974595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:32:55.974539    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:00.722236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:00.722206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:05.384980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:05.384938    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:05.549270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:05.549228    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:05.690095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:05.689987    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:10.391474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.391436    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:10.498039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.498000    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"]
Nov 01 09:33:10.502859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.502835    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.557258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.557220    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"]
Nov 01 09:33:10.597283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.597243    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:33:10.597471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.597301    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:33:10.597471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.597302    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:33:10.601261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.601231    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:33:10.655124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.655026    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.655124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.655074    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gkg\" (UniqueName: \"kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.655413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.655383    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.757060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.757013    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.757272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.757099    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.757272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.757129    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gkg\" (UniqueName: \"kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.757914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.757887    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.759565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.759525    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.814629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.814597    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gkg\" (UniqueName: \"kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg\") pod \"maas-api-revoke-sub-e652ddf7-m9vzg\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:10.819433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:10.819400    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:11.203210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:11.203171    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"]
Nov 01 09:33:11.203788 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:33:11.203757    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af0e5af_0c95_4a87_a7ae_90ae8f0d3aa2.slice/crio-c63605aa0fe0783130029b2c8e75fcf51b4f813129f89567a9e19310434d208b WatchSource:0}: Error finding container c63605aa0fe0783130029b2c8e75fcf51b4f813129f89567a9e19310434d208b: Status 404 returned error can't find the container with id c63605aa0fe0783130029b2c8e75fcf51b4f813129f89567a9e19310434d208b
Nov 01 09:33:11.327058 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:33:11.327021    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:33:11.522129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:11.522090    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg" event={"ID":"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2","Type":"ContainerStarted","Data":"f841dd452436185d22db45fab22147fceeadde933253b03789f99e17fd4f73bb"}
Nov 01 09:33:11.522665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:11.522187    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg" event={"ID":"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2","Type":"ContainerStarted","Data":"c63605aa0fe0783130029b2c8e75fcf51b4f813129f89567a9e19310434d208b"}
Nov 01 09:33:11.630262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:11.630184    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg" podStartSLOduration=1.6301674849999999 podStartE2EDuration="1.630167485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:11.629472449 +0000 UTC m=+3664.823867007" watchObservedRunningTime="2026-11-01 09:33:11.630167485 +0000 UTC m=+3664.824562046"
Nov 01 09:33:12.530916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:12.530880    4314 generic.go:356] "Generic (PLEG): container finished" podID="5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" containerID="f841dd452436185d22db45fab22147fceeadde933253b03789f99e17fd4f73bb" exitCode=0
Nov 01 09:33:12.531391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:12.530965    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg" event={"ID":"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2","Type":"ContainerDied","Data":"f841dd452436185d22db45fab22147fceeadde933253b03789f99e17fd4f73bb"}
Nov 01 09:33:13.703457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.703429    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:13.896162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.896081    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg\" (UniqueName: \"kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg\") pod \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") "
Nov 01 09:33:13.896317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.896246    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca\") pod \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") "
Nov 01 09:33:13.896409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.896392    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca\") pod \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\" (UID: \"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2\") "
Nov 01 09:33:13.896642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.896616    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca" pod "5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" (UID: "5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:33:13.896905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.896880    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:13.898409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.898384    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca" pod "5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" (UID: "5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:33:13.898508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.898431    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg" pod "5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" (UID: "5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2"). InnerVolumeSpecName "kube-api-access-48gkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:33:13.998303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.998270    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:13.998303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:13.998300    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gkg\" (UniqueName: \"kubernetes.io/projected/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2-kube-api-access-48gkg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:14.545936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:14.545901    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"
Nov 01 09:33:14.546133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:14.545933    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg" event={"ID":"5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2","Type":"ContainerDied","Data":"c63605aa0fe0783130029b2c8e75fcf51b4f813129f89567a9e19310434d208b"}
Nov 01 09:33:14.546133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:14.545986    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c63605aa0fe0783130029b2c8e75fcf51b4f813129f89567a9e19310434d208b"
Nov 01 09:33:15.287252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:15.287217    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:20.089224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:20.089184    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:20.898203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:20.898172    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:21.354396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.354359    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"]
Nov 01 09:33:21.355053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.355037    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" containerName="revoke-keys"
Nov 01 09:33:21.355114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.355055    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" containerName="revoke-keys"
Nov 01 09:33:21.355215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.355205    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" containerName="revoke-keys"
Nov 01 09:33:21.358859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.358844    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.393594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.393558    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98h9v\" (UniqueName: \"kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.393774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.393699    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.393819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.393806    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.439465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.439428    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:33:21.439465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.439445    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:33:21.443562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.443506    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"]
Nov 01 09:33:21.454498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.454473    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:33:21.454498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.454489    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:33:21.494909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.494875    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.495158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.495001    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98h9v\" (UniqueName: \"kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.495158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.495139    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.495736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.495715    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.497420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.497396    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.552020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.551979    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98h9v\" (UniqueName: \"kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v\") pod \"maas-api-revoke-sub-3187b904-hg49g\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:21.673141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:21.673061    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:22.049730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:22.049699    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"]
Nov 01 09:33:22.049993 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:33:22.049968    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801eb4f6_5acf_4720_a0fc_3d84c7127d68.slice/crio-b4fdd48c59c72372d2a6e433016512d176f0481d089206325dfdf9556a3c0bd6 WatchSource:0}: Error finding container b4fdd48c59c72372d2a6e433016512d176f0481d089206325dfdf9556a3c0bd6: Status 404 returned error can't find the container with id b4fdd48c59c72372d2a6e433016512d176f0481d089206325dfdf9556a3c0bd6
Nov 01 09:33:22.619465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:22.619368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g" event={"ID":"801eb4f6-5acf-4720-a0fc-3d84c7127d68","Type":"ContainerStarted","Data":"d7e20145930c782bfba3f1c59456640f00ac7cda10a7b13df70656919699cde6"}
Nov 01 09:33:22.619465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:22.619421    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g" event={"ID":"801eb4f6-5acf-4720-a0fc-3d84c7127d68","Type":"ContainerStarted","Data":"b4fdd48c59c72372d2a6e433016512d176f0481d089206325dfdf9556a3c0bd6"}
Nov 01 09:33:22.729198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:22.729142    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g" podStartSLOduration=1.72912552 podStartE2EDuration="1.72912552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:22.726467507 +0000 UTC m=+3675.920862066" watchObservedRunningTime="2026-11-01 09:33:22.72912552 +0000 UTC m=+3675.923520074"
Nov 01 09:33:23.627671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:23.627636    4314 generic.go:356] "Generic (PLEG): container finished" podID="801eb4f6-5acf-4720-a0fc-3d84c7127d68" containerID="d7e20145930c782bfba3f1c59456640f00ac7cda10a7b13df70656919699cde6" exitCode=0
Nov 01 09:33:23.628097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:23.627686    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g" event={"ID":"801eb4f6-5acf-4720-a0fc-3d84c7127d68","Type":"ContainerDied","Data":"d7e20145930c782bfba3f1c59456640f00ac7cda10a7b13df70656919699cde6"}
Nov 01 09:33:23.933267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:23.933183    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:24.801588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.801561    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:24.942739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.942667    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca\") pod \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") "
Nov 01 09:33:24.942911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.942823    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v\" (UniqueName: \"kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v\") pod \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") "
Nov 01 09:33:24.943032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.943016    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca\") pod \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\" (UID: \"801eb4f6-5acf-4720-a0fc-3d84c7127d68\") "
Nov 01 09:33:24.943129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.943087    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca" pod "801eb4f6-5acf-4720-a0fc-3d84c7127d68" (UID: "801eb4f6-5acf-4720-a0fc-3d84c7127d68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:33:24.943447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.943427    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:24.945421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.945398    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v" pod "801eb4f6-5acf-4720-a0fc-3d84c7127d68" (UID: "801eb4f6-5acf-4720-a0fc-3d84c7127d68"). InnerVolumeSpecName "kube-api-access-98h9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:33:24.945954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:24.945930    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca" pod "801eb4f6-5acf-4720-a0fc-3d84c7127d68" (UID: "801eb4f6-5acf-4720-a0fc-3d84c7127d68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:33:25.044776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.044741    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98h9v\" (UniqueName: \"kubernetes.io/projected/801eb4f6-5acf-4720-a0fc-3d84c7127d68-kube-api-access-98h9v\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:25.044776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.044769    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/801eb4f6-5acf-4720-a0fc-3d84c7127d68-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:25.159459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.159417    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:25.335805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.335766    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:25.644455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.644380    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"
Nov 01 09:33:25.644455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.644382    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g" event={"ID":"801eb4f6-5acf-4720-a0fc-3d84c7127d68","Type":"ContainerDied","Data":"b4fdd48c59c72372d2a6e433016512d176f0481d089206325dfdf9556a3c0bd6"}
Nov 01 09:33:25.644455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.644434    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4fdd48c59c72372d2a6e433016512d176f0481d089206325dfdf9556a3c0bd6"
Nov 01 09:33:25.967028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:25.966934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:26.241276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:26.241232    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:29.977585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:29.977520    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:30.166947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:30.166912    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:30.312324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:30.312288    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:30.454469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:30.454439    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:33.111303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.111270    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:33.271733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.271701    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:33.512543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.512492    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"]
Nov 01 09:33:33.513158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.513143    4314 cpu_manager.go:414] "containerMap: removing container" podUID="801eb4f6-5acf-4720-a0fc-3d84c7127d68" containerName="revoke-keys"
Nov 01 09:33:33.513207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.513160    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801eb4f6-5acf-4720-a0fc-3d84c7127d68" containerName="revoke-keys"
Nov 01 09:33:33.513283 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.513274    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="801eb4f6-5acf-4720-a0fc-3d84c7127d68" containerName="revoke-keys"
Nov 01 09:33:33.517380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.517360    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.571052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.571022    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"]
Nov 01 09:33:33.611768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.611736    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b0d0f3\""
Nov 01 09:33:33.611966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.611863    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b0d0f3-dockercfg-qlb5x\""
Nov 01 09:33:33.705223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.705183    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbdr\" (UniqueName: \"kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr\") pod \"payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.705435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.705415    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.710388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.710351    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"]
Nov 01 09:33:33.715865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.715837    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:33.741701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.741671    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"]
Nov 01 09:33:33.806329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.806239    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbdr\" (UniqueName: \"kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr\") pod \"payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.806329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.806285    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqhd\" (UniqueName: \"kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd\") pod \"payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:33.806609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.806487    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:33.806609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.806564    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.807242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.807222    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.863299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.863265    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbdr\" (UniqueName: \"kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr\") pod \"payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:33.907875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.907835    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqhd\" (UniqueName: \"kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd\") pod \"payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:33.908074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.908012    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:33.908789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.908761    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:33.965181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:33.965148    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqhd\" (UniqueName: \"kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd\") pod \"payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:34.033870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.033830    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:34.131691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.131625    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:34.406474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.406443    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"]
Nov 01 09:33:34.407115 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:33:34.407090    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f197e1f_d112_4c5c_9036_ca459b164b45.slice/crio-e7a8df9a8c3ebced48243c7608a5638a36a0bf21ee46375c8ca0530ce3708e80 WatchSource:0}: Error finding container e7a8df9a8c3ebced48243c7608a5638a36a0bf21ee46375c8ca0530ce3708e80: Status 404 returned error can't find the container with id e7a8df9a8c3ebced48243c7608a5638a36a0bf21ee46375c8ca0530ce3708e80
Nov 01 09:33:34.513666 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:33:34.513639    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d70813_2012_423b_bb37_96b0138fa9d1.slice/crio-5fef2d6c5f58c94027e8986409de5b519821c70ee43297a7d47cfd77d7d65124 WatchSource:0}: Error finding container 5fef2d6c5f58c94027e8986409de5b519821c70ee43297a7d47cfd77d7d65124: Status 404 returned error can't find the container with id 5fef2d6c5f58c94027e8986409de5b519821c70ee43297a7d47cfd77d7d65124
Nov 01 09:33:34.513902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.513877    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"]
Nov 01 09:33:34.717581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.717461    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72" event={"ID":"8f197e1f-d112-4c5c-9036-ca459b164b45","Type":"ContainerStarted","Data":"4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92"}
Nov 01 09:33:34.717581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.717551    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:34.717581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.717575    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72" event={"ID":"8f197e1f-d112-4c5c-9036-ca459b164b45","Type":"ContainerStarted","Data":"e7a8df9a8c3ebced48243c7608a5638a36a0bf21ee46375c8ca0530ce3708e80"}
Nov 01 09:33:34.719167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.719141    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb" event={"ID":"01d70813-2012-423b-bb37-96b0138fa9d1","Type":"ContainerStarted","Data":"830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602"}
Nov 01 09:33:34.719502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.719184    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb" event={"ID":"01d70813-2012-423b-bb37-96b0138fa9d1","Type":"ContainerStarted","Data":"5fef2d6c5f58c94027e8986409de5b519821c70ee43297a7d47cfd77d7d65124"}
Nov 01 09:33:34.719502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.719226    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:34.829160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.828295    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72" podStartSLOduration=1.828269139 podStartE2EDuration="1.828269139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:34.825448134 +0000 UTC m=+3688.019842702" watchObservedRunningTime="2026-11-01 09:33:34.828269139 +0000 UTC m=+3688.022663696"
Nov 01 09:33:34.937890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:34.937833    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb" podStartSLOduration=1.9378180839999999 podStartE2EDuration="1.937818084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:34.935261199 +0000 UTC m=+3688.129655744" watchObservedRunningTime="2026-11-01 09:33:34.937818084 +0000 UTC m=+3688.132212639"
Nov 01 09:33:35.364257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:35.364226    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"]
Nov 01 09:33:35.401966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:35.401934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:35.560614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:35.560580    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:36.799852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:36.799815    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"]
Nov 01 09:33:36.827166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:36.827132    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:37.531374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:37.531342    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:38.022241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.022209    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"]
Nov 01 09:33:38.026704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.026678    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059130    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059172    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059200    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059313    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059340    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059365    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059437    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xmb\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059484    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.059630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.059558    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.075270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.075243    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1f3121-openshift-default-dockercfg-nc2b7\""
Nov 01 09:33:38.125875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.125844    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"]
Nov 01 09:33:38.127462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.127430    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:38.161195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161144    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xmb\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161214    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161270    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161344    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161368    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161393    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161467    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161496    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161549    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161694    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.161753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.161715    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.162094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.162070    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.162459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.162348    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.162459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.162401    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.164749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.164700    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.165060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.165038    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.218670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.218639    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.218829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.218711    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xmb\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb\") pod \"e2e-autores-1f3121-openshift-default-797c788959-klwt4\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") " pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.296180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.296104    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:38.341420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.341388    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:38.764217 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:33:38.764188    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb60f2c_7c8d_438b_bde4_45f9034fce60.slice/crio-6b32f60326b94f4401f0a91f6e4dab26f313b80b491511ec44a91e4a845d5e6b WatchSource:0}: Error finding container 6b32f60326b94f4401f0a91f6e4dab26f313b80b491511ec44a91e4a845d5e6b: Status 404 returned error can't find the container with id 6b32f60326b94f4401f0a91f6e4dab26f313b80b491511ec44a91e4a845d5e6b
Nov 01 09:33:38.764732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.764702    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"]
Nov 01 09:33:38.767351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.767318    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:33:38.767453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.767377    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:33:38.767453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:38.767407    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:33:39.756257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:39.756220    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" event={"ID":"4eb60f2c-7c8d-438b-bde4-45f9034fce60","Type":"ContainerStarted","Data":"31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"}
Nov 01 09:33:39.756721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:39.756319    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" event={"ID":"4eb60f2c-7c8d-438b-bde4-45f9034fce60","Type":"ContainerStarted","Data":"6b32f60326b94f4401f0a91f6e4dab26f313b80b491511ec44a91e4a845d5e6b"}
Nov 01 09:33:39.918623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:39.918564    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podStartSLOduration=2.91855027 podStartE2EDuration="2.91855027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:39.915295218 +0000 UTC m=+3693.109689772" watchObservedRunningTime="2026-11-01 09:33:39.91855027 +0000 UTC m=+3693.112944825"
Nov 01 09:33:40.342166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:40.342130    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:40.348115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:40.348087    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:40.763527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:40.763498    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:40.764511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:40.764486    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:33:40.935174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:40.935123    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:41.324359 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:33:41.324324    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:33:43.354365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:43.354332    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:43.531696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:43.531665    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:43.681319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:43.681227    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:45.729616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:45.729583    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:33:45.730023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:45.729658    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:33:46.210178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:46.210121    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:46.495096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:46.495066    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:46.684362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:46.684326    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:50.385942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.385888    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6" podUID="b67eb486-8b20-49ab-91e4-a621e4c4605b" containerName="main" containerID="cri-o://6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8" gracePeriod=45
Nov 01 09:33:50.768599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.768568    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:33:50.861496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.861460    4314 generic.go:356] "Generic (PLEG): container finished" podID="b67eb486-8b20-49ab-91e4-a621e4c4605b" containerID="6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8" exitCode=0
Nov 01 09:33:50.861718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.861521    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"
Nov 01 09:33:50.861718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.861527    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6" event={"ID":"b67eb486-8b20-49ab-91e4-a621e4c4605b","Type":"ContainerDied","Data":"6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8"}
Nov 01 09:33:50.861718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.861673    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6" event={"ID":"b67eb486-8b20-49ab-91e4-a621e4c4605b","Type":"ContainerDied","Data":"7e9e83b1a8e3b0474af7a4f7ddb2c57542ec8f90a53d4058a35fc340cbe58ee0"}
Nov 01 09:33:50.861718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.861701    4314 scope.go:122] "RemoveContainer" containerID="6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8"
Nov 01 09:33:50.868953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.868925    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home\") pod \"b67eb486-8b20-49ab-91e4-a621e4c4605b\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") "
Nov 01 09:33:50.869141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869122    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs\") pod \"b67eb486-8b20-49ab-91e4-a621e4c4605b\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") "
Nov 01 09:33:50.869240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869222    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache\") pod \"b67eb486-8b20-49ab-91e4-a621e4c4605b\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") "
Nov 01 09:33:50.869303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869256    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm\") pod \"b67eb486-8b20-49ab-91e4-a621e4c4605b\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") "
Nov 01 09:33:50.869303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869284    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir\") pod \"b67eb486-8b20-49ab-91e4-a621e4c4605b\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") "
Nov 01 09:33:50.869423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869331    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n\" (UniqueName: \"kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n\") pod \"b67eb486-8b20-49ab-91e4-a621e4c4605b\" (UID: \"b67eb486-8b20-49ab-91e4-a621e4c4605b\") "
Nov 01 09:33:50.869423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869380    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home" pod "b67eb486-8b20-49ab-91e4-a621e4c4605b" (UID: "b67eb486-8b20-49ab-91e4-a621e4c4605b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:50.869671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869646    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache" pod "b67eb486-8b20-49ab-91e4-a621e4c4605b" (UID: "b67eb486-8b20-49ab-91e4-a621e4c4605b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:50.870105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869893    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:50.870105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.869918    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:50.870105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.870075    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir" pod "b67eb486-8b20-49ab-91e4-a621e4c4605b" (UID: "b67eb486-8b20-49ab-91e4-a621e4c4605b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:50.872912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.872872    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n" pod "b67eb486-8b20-49ab-91e4-a621e4c4605b" (UID: "b67eb486-8b20-49ab-91e4-a621e4c4605b"). InnerVolumeSpecName "kube-api-access-2w88n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:33:50.875997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.875973    4314 scope.go:122] "RemoveContainer" containerID="6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8"
Nov 01 09:33:50.876455 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:33:50.876375    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8\": container with ID starting with 6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8 not found: ID does not exist" containerID="6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8"
Nov 01 09:33:50.876455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.876415    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8"} err="failed to get container status \"6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8\": rpc error: code = NotFound desc = could not find container \"6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8\": container with ID starting with 6b97b45ff13eebdb18a01520ede27124c9702a2f49680107fc9083fd239ec5c8 not found: ID does not exist"
Nov 01 09:33:50.884027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.883973    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs" pod "b67eb486-8b20-49ab-91e4-a621e4c4605b" (UID: "b67eb486-8b20-49ab-91e4-a621e4c4605b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:33:50.890646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.890586    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm" pod "b67eb486-8b20-49ab-91e4-a621e4c4605b" (UID: "b67eb486-8b20-49ab-91e4-a621e4c4605b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:50.971777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.971738    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67eb486-8b20-49ab-91e4-a621e4c4605b-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:50.971777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.971775    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:50.971777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.971786    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67eb486-8b20-49ab-91e4-a621e4c4605b-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:50.972044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:50.971799    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w88n\" (UniqueName: \"kubernetes.io/projected/b67eb486-8b20-49ab-91e4-a621e4c4605b-kube-api-access-2w88n\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:51.271365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:51.271319    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"]
Nov 01 09:33:51.317566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:51.317519    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-58de5a/auth-test-model-58de5a-kserve-957cf54fc-xwlw6"]
Nov 01 09:33:51.522856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:51.522778    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:51.793798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:51.793704    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:33:51.793798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:51.793776    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:33:51.820693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:51.820650    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8" podUID="8499201a-74b6-4f23-b97f-66a43f5df7d4" containerName="main" containerID="cri-o://66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881" gracePeriod=45
Nov 01 09:33:52.099748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.099721    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:33:52.186585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186515    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs\") pod \"8499201a-74b6-4f23-b97f-66a43f5df7d4\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") "
Nov 01 09:33:52.186795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186618    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home\") pod \"8499201a-74b6-4f23-b97f-66a43f5df7d4\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") "
Nov 01 09:33:52.186795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186688    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir\") pod \"8499201a-74b6-4f23-b97f-66a43f5df7d4\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") "
Nov 01 09:33:52.186795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186759    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm\") pod \"8499201a-74b6-4f23-b97f-66a43f5df7d4\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") "
Nov 01 09:33:52.186981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186800    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m\" (UniqueName: \"kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m\") pod \"8499201a-74b6-4f23-b97f-66a43f5df7d4\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") "
Nov 01 09:33:52.186981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186900    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache\") pod \"8499201a-74b6-4f23-b97f-66a43f5df7d4\" (UID: \"8499201a-74b6-4f23-b97f-66a43f5df7d4\") "
Nov 01 09:33:52.186981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.186920    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home" pod "8499201a-74b6-4f23-b97f-66a43f5df7d4" (UID: "8499201a-74b6-4f23-b97f-66a43f5df7d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:52.187151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.187001    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir" pod "8499201a-74b6-4f23-b97f-66a43f5df7d4" (UID: "8499201a-74b6-4f23-b97f-66a43f5df7d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:52.187324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.187268    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache" pod "8499201a-74b6-4f23-b97f-66a43f5df7d4" (UID: "8499201a-74b6-4f23-b97f-66a43f5df7d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:52.187605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.187548    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:52.187605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.187568    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:52.187605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.187580    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:52.189436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.189406    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m" pod "8499201a-74b6-4f23-b97f-66a43f5df7d4" (UID: "8499201a-74b6-4f23-b97f-66a43f5df7d4"). InnerVolumeSpecName "kube-api-access-bbn7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:33:52.189568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.189489    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs" pod "8499201a-74b6-4f23-b97f-66a43f5df7d4" (UID: "8499201a-74b6-4f23-b97f-66a43f5df7d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:33:52.189629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.189597    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm" pod "8499201a-74b6-4f23-b97f-66a43f5df7d4" (UID: "8499201a-74b6-4f23-b97f-66a43f5df7d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:33:52.288909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.288868    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499201a-74b6-4f23-b97f-66a43f5df7d4-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:52.288909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.288906    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbn7m\" (UniqueName: \"kubernetes.io/projected/8499201a-74b6-4f23-b97f-66a43f5df7d4-kube-api-access-bbn7m\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:52.289172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.288937    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499201a-74b6-4f23-b97f-66a43f5df7d4-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:52.877342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.877307    4314 generic.go:356] "Generic (PLEG): container finished" podID="8499201a-74b6-4f23-b97f-66a43f5df7d4" containerID="66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881" exitCode=0
Nov 01 09:33:52.877809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.877348    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"
Nov 01 09:33:52.877809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.877395    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8" event={"ID":"8499201a-74b6-4f23-b97f-66a43f5df7d4","Type":"ContainerDied","Data":"66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881"}
Nov 01 09:33:52.877809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.877438    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8" event={"ID":"8499201a-74b6-4f23-b97f-66a43f5df7d4","Type":"ContainerDied","Data":"3da2ae3d4bf2ba57e40737859b909e4fbc7ec18bce8f208ca1010b32b3dc257d"}
Nov 01 09:33:52.877809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.877459    4314 scope.go:122] "RemoveContainer" containerID="66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881"
Nov 01 09:33:52.890277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.890250    4314 scope.go:122] "RemoveContainer" containerID="66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881"
Nov 01 09:33:52.890678 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:33:52.890656    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881\": container with ID starting with 66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881 not found: ID does not exist" containerID="66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881"
Nov 01 09:33:52.890803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.890686    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881"} err="failed to get container status \"66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881\": rpc error: code = NotFound desc = could not find container \"66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881\": container with ID starting with 66f0a4198a7b04ac232b79ef9f697c49938bd24566741b6267e37be4c1490881 not found: ID does not exist"
Nov 01 09:33:52.929662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:52.929632    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67eb486-8b20-49ab-91e4-a621e4c4605b" path="/var/lib/kubelet/pods/b67eb486-8b20-49ab-91e4-a621e4c4605b/volumes"
Nov 01 09:33:53.032753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:53.032711    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"]
Nov 01 09:33:53.078667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:53.078635    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a83714/auth-test-model-a83714-kserve-559f7c59bf-bc4c8"]
Nov 01 09:33:54.900206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.900171    4314 generic.go:356] "Generic (PLEG): container finished" podID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerID="0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383" exitCode=0
Nov 01 09:33:54.900695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.900241    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" event={"ID":"59a5bfcf-d256-4430-99b9-8c5c3d599f4c","Type":"ContainerDied","Data":"0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"}
Nov 01 09:33:54.900747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.900727    4314 scope.go:122] "RemoveContainer" containerID="0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"
Nov 01 09:33:54.902481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.902452    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:33:54.902580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.902523    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:33:54.902623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.902581    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:33:54.929843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:54.929814    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8499201a-74b6-4f23-b97f-66a43f5df7d4" path="/var/lib/kubelet/pods/8499201a-74b6-4f23-b97f-66a43f5df7d4/volumes"
Nov 01 09:33:55.147779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:55.147745    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:55.909208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:55.909174    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" event={"ID":"59a5bfcf-d256-4430-99b9-8c5c3d599f4c","Type":"ContainerStarted","Data":"2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415"}
Nov 01 09:33:56.664012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:56.663936    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:33:56.826541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:56.826492    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:33:56.826759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:56.826692    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 01 09:33:56.826843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:56.826749    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 01 09:33:57.826977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:57.826941    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 01 09:33:57.827407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:57.826998    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 01 09:33:58.826765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:58.826725    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 01 09:33:58.826993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:58.826800    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 01 09:33:59.084394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:59.084283    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"]
Nov 01 09:33:59.088670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:59.088633    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54354543-c74mm"]
Nov 01 09:33:59.826391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:59.826354    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 01 09:33:59.826684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:33:59.826422    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 01 09:34:00.284141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.284094    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"]
Nov 01 09:34:00.385641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.385602    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"]
Nov 01 09:34:00.607214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.607133    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:00.778557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.774454    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:00.826658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.826618    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 01 09:34:00.826885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.826684    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 01 09:34:00.930667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:00.930585    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f353989a-652c-41ad-8d3f-1b1f517742e3" path="/var/lib/kubelet/pods/f353989a-652c-41ad-8d3f-1b1f517742e3/volumes"
Nov 01 09:34:01.747550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:01.747485    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:01.830907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:01.830877    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:34:01.831346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:01.831325    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:34:01.832152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:01.832132    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:34:01.906635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:01.906598    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:05.290275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.290049    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp" podUID="c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" containerName="payload-processing" containerID="cri-o://76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05" gracePeriod=25
Nov 01 09:34:05.386250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.386168    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2" podUID="8c67b251-a132-46ff-82eb-11985fe5814f" containerName="payload-pre-processing" containerID="cri-o://05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc" gracePeriod=25
Nov 01 09:34:05.669994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.669962    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:34:05.697327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.697303    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:34:05.804446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.804367    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k\" (UniqueName: \"kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k\") pod \"8c67b251-a132-46ff-82eb-11985fe5814f\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") "
Nov 01 09:34:05.804446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.804436    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume\") pod \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") "
Nov 01 09:34:05.804674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.804625    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume\") pod \"8c67b251-a132-46ff-82eb-11985fe5814f\" (UID: \"8c67b251-a132-46ff-82eb-11985fe5814f\") "
Nov 01 09:34:05.804829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.804812    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh\" (UniqueName: \"kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh\") pod \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\" (UID: \"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc\") "
Nov 01 09:34:05.805000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.804977    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume" pod "c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" (UID: "c493ba1b-ba21-400a-b40a-9bea6cc5f7fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:34:05.805097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.805041    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume" pod "8c67b251-a132-46ff-82eb-11985fe5814f" (UID: "8c67b251-a132-46ff-82eb-11985fe5814f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:34:05.805401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.805378    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c67b251-a132-46ff-82eb-11985fe5814f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:05.805500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.805406    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:05.806906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.806882    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh" pod "c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" (UID: "c493ba1b-ba21-400a-b40a-9bea6cc5f7fc"). InnerVolumeSpecName "kube-api-access-bsnjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:34:05.807149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.807131    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k" pod "8c67b251-a132-46ff-82eb-11985fe5814f" (UID: "8c67b251-a132-46ff-82eb-11985fe5814f"). InnerVolumeSpecName "kube-api-access-c9q9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:34:05.906918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.906886    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnjh\" (UniqueName: \"kubernetes.io/projected/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc-kube-api-access-bsnjh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:05.906918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.906915    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9q9k\" (UniqueName: \"kubernetes.io/projected/8c67b251-a132-46ff-82eb-11985fe5814f-kube-api-access-c9q9k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:05.982245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.982208    4314 generic.go:356] "Generic (PLEG): container finished" podID="c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" containerID="76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05" exitCode=0
Nov 01 09:34:05.982430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.982262    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"
Nov 01 09:34:05.982430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.982295    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp" event={"ID":"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc","Type":"ContainerDied","Data":"76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05"}
Nov 01 09:34:05.982430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.982352    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp" event={"ID":"c493ba1b-ba21-400a-b40a-9bea6cc5f7fc","Type":"ContainerDied","Data":"ed1e9d599e06595e5a6860efe9e0653273945c5c6178ec390080d87f12b22d15"}
Nov 01 09:34:05.982430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.982376    4314 scope.go:122] "RemoveContainer" containerID="76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05"
Nov 01 09:34:05.983909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.983886    4314 generic.go:356] "Generic (PLEG): container finished" podID="8c67b251-a132-46ff-82eb-11985fe5814f" containerID="05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc" exitCode=0
Nov 01 09:34:05.984048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.983908    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2" event={"ID":"8c67b251-a132-46ff-82eb-11985fe5814f","Type":"ContainerDied","Data":"05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc"}
Nov 01 09:34:05.984048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.983930    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"
Nov 01 09:34:05.984048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.983954    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2" event={"ID":"8c67b251-a132-46ff-82eb-11985fe5814f","Type":"ContainerDied","Data":"cf9d51fa809da1121f88d223ffceb82f9de2a427e91a14207cd631555865d740"}
Nov 01 09:34:05.993237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.993218    4314 scope.go:122] "RemoveContainer" containerID="76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05"
Nov 01 09:34:05.993565 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:34:05.993518    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05\": container with ID starting with 76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05 not found: ID does not exist" containerID="76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05"
Nov 01 09:34:05.993684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.993565    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05"} err="failed to get container status \"76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05\": rpc error: code = NotFound desc = could not find container \"76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05\": container with ID starting with 76001c99288637b3c94ab93d34f3b283bd5d3f170e40b98ca18a47f4f71dbc05 not found: ID does not exist"
Nov 01 09:34:05.993684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:05.993585    4314 scope.go:122] "RemoveContainer" containerID="05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc"
Nov 01 09:34:06.002801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.002783    4314 scope.go:122] "RemoveContainer" containerID="05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc"
Nov 01 09:34:06.003078 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:34:06.003058    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc\": container with ID starting with 05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc not found: ID does not exist" containerID="05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc"
Nov 01 09:34:06.003163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.003085    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc"} err="failed to get container status \"05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc\": rpc error: code = NotFound desc = could not find container \"05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc\": container with ID starting with 05a8663a3732734d59a9c88bb345178ec9b2ace093698d17430306557d560bcc not found: ID does not exist"
Nov 01 09:34:06.137583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.137475    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"]
Nov 01 09:34:06.145360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.145327    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25d9fe-778ff97c4c-vpcg2"]
Nov 01 09:34:06.250726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.250693    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"]
Nov 01 09:34:06.263965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.263935    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25d9fe-78f84599cc-n4lmp"]
Nov 01 09:34:06.796184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.796145    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:06.931069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.931035    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c67b251-a132-46ff-82eb-11985fe5814f" path="/var/lib/kubelet/pods/8c67b251-a132-46ff-82eb-11985fe5814f/volumes"
Nov 01 09:34:06.931499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:06.931483    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" path="/var/lib/kubelet/pods/c493ba1b-ba21-400a-b40a-9bea6cc5f7fc/volumes"
Nov 01 09:34:07.362460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:07.362420    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:08.748636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:08.748604    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:09.047273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:09.047188    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"]
Nov 01 09:34:09.050790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:09.050766    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecb47ab-4dtxt"]
Nov 01 09:34:10.931298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:10.931258    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46de1d0e-0f0c-437e-9e55-03acf5373160" path="/var/lib/kubelet/pods/46de1d0e-0f0c-437e-9e55-03acf5373160/volumes"
Nov 01 09:34:11.326250 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:34:11.326207    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:34:11.673624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:11.673509    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:14.627732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:14.627695    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:14.974435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:14.974351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:15.126446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:15.126396    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:16.503482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:16.503443    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:20.379765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:20.379667    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:21.287620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:21.287586    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:21.432523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:21.432479    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:26.099710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:26.099677    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:30.880126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:30.880094    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:31.962609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:31.962573    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:35.610021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:35.609968    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:35.754704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:35.754663    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:40.387707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:40.387667    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:41.326560 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:34:41.326511    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:34:41.835300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:41.835266    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:42.635636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:42.635607    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"]
Nov 01 09:34:42.734032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:42.733999    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"]
Nov 01 09:34:42.979250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:42.979169    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:43.120493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:43.120450    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:45.407674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:45.407639    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:47.612135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:47.612090    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:47.636484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:47.636451    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7" podUID="36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" containerName="payload-processing" containerID="cri-o://45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6" gracePeriod=25
Nov 01 09:34:47.735106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:47.735071    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4" podUID="6cb99506-a507-4313-86c6-9b47e9ed920d" containerName="payload-pre-processing" containerID="cri-o://7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916" gracePeriod=25
Nov 01 09:34:47.906683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:47.906652    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:34:48.007915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.007891    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume\") pod \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") "
Nov 01 09:34:48.008059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.007992    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7\" (UniqueName: \"kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7\") pod \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\" (UID: \"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d\") "
Nov 01 09:34:48.008265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.008241    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume" pod "36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" (UID: "36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:34:48.008656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.008636    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:48.010102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.010077    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7" pod "36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" (UID: "36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d"). InnerVolumeSpecName "kube-api-access-dlbf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:34:48.017162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.017138    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:34:48.110140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.110106    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume\") pod \"6cb99506-a507-4313-86c6-9b47e9ed920d\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") "
Nov 01 09:34:48.110140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.110143    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5\" (UniqueName: \"kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5\") pod \"6cb99506-a507-4313-86c6-9b47e9ed920d\" (UID: \"6cb99506-a507-4313-86c6-9b47e9ed920d\") "
Nov 01 09:34:48.110474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.110454    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbf7\" (UniqueName: \"kubernetes.io/projected/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d-kube-api-access-dlbf7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:48.110474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.110455    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume" pod "6cb99506-a507-4313-86c6-9b47e9ed920d" (UID: "6cb99506-a507-4313-86c6-9b47e9ed920d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:34:48.112299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.112270    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5" pod "6cb99506-a507-4313-86c6-9b47e9ed920d" (UID: "6cb99506-a507-4313-86c6-9b47e9ed920d"). InnerVolumeSpecName "kube-api-access-czzk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:34:48.212175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.212094    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb99506-a507-4313-86c6-9b47e9ed920d-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:48.212175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.212123    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzk5\" (UniqueName: \"kubernetes.io/projected/6cb99506-a507-4313-86c6-9b47e9ed920d-kube-api-access-czzk5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:48.304523 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.304488    4314 generic.go:356] "Generic (PLEG): container finished" podID="6cb99506-a507-4313-86c6-9b47e9ed920d" containerID="7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916" exitCode=0
Nov 01 09:34:48.304727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.304561    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"
Nov 01 09:34:48.304727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.304572    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4" event={"ID":"6cb99506-a507-4313-86c6-9b47e9ed920d","Type":"ContainerDied","Data":"7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916"}
Nov 01 09:34:48.304727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.304636    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4" event={"ID":"6cb99506-a507-4313-86c6-9b47e9ed920d","Type":"ContainerDied","Data":"90f904ff08ead6d64e58616ed8bcdbde26f5dc9eda148dd677a9932ec181c47b"}
Nov 01 09:34:48.304727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.304668    4314 scope.go:122] "RemoveContainer" containerID="7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916"
Nov 01 09:34:48.306384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.306358    4314 generic.go:356] "Generic (PLEG): container finished" podID="36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" containerID="45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6" exitCode=0
Nov 01 09:34:48.306614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.306395    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7" event={"ID":"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d","Type":"ContainerDied","Data":"45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6"}
Nov 01 09:34:48.306614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.306425    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"
Nov 01 09:34:48.306614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.306455    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7" event={"ID":"36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d","Type":"ContainerDied","Data":"f76ca11bd0db4879898baf4d49e13c19f63002657aef6edb0d3e3a210802b7ec"}
Nov 01 09:34:48.316895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.316870    4314 scope.go:122] "RemoveContainer" containerID="7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916"
Nov 01 09:34:48.317171 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:34:48.317146    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916\": container with ID starting with 7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916 not found: ID does not exist" containerID="7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916"
Nov 01 09:34:48.317270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.317180    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916"} err="failed to get container status \"7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916\": rpc error: code = NotFound desc = could not find container \"7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916\": container with ID starting with 7eae87a0f7289730cdce9968d16042e73f5e2f0a390af431ab68bef73990e916 not found: ID does not exist"
Nov 01 09:34:48.317270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.317206    4314 scope.go:122] "RemoveContainer" containerID="45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6"
Nov 01 09:34:48.329948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.329924    4314 scope.go:122] "RemoveContainer" containerID="45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6"
Nov 01 09:34:48.330239 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:34:48.330217    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6\": container with ID starting with 45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6 not found: ID does not exist" containerID="45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6"
Nov 01 09:34:48.330301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.330250    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6"} err="failed to get container status \"45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6\": rpc error: code = NotFound desc = could not find container \"45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6\": container with ID starting with 45625993d321b34b5716459d7571b6b8fc9bbb381d9b617e6b01e8f1c1d3e2e6 not found: ID does not exist"
Nov 01 09:34:48.471458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.471375    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"]
Nov 01 09:34:48.475113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.475084    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-54c25e-7c45f87777-28mt7"]
Nov 01 09:34:48.575723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.575681    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"]
Nov 01 09:34:48.580095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.580066    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-54c25e-65f74d5c75-vfql4"]
Nov 01 09:34:48.930960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.930933    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" path="/var/lib/kubelet/pods/36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d/volumes"
Nov 01 09:34:48.931346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:48.931315    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb99506-a507-4313-86c6-9b47e9ed920d" path="/var/lib/kubelet/pods/6cb99506-a507-4313-86c6-9b47e9ed920d/volumes"
Nov 01 09:34:49.048242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:49.048210    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"]
Nov 01 09:34:49.054986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:49.054958    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40c94d39-c6ccb"]
Nov 01 09:34:50.486281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:50.486244    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:34:50.929713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:50.929638    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99627d1-4d31-4b0c-a031-943afa967ad1" path="/var/lib/kubelet/pods/a99627d1-4d31-4b0c-a031-943afa967ad1/volumes"
Nov 01 09:34:51.045869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:51.045835    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"]
Nov 01 09:34:51.093014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:51.092977    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db45573-hf25j"]
Nov 01 09:34:52.929887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:52.929852    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b292ca-98a5-45f8-9c89-8f6be656b3c6" path="/var/lib/kubelet/pods/c0b292ca-98a5-45f8-9c89-8f6be656b3c6/volumes"
Nov 01 09:34:55.281044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:34:55.281012    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:00.047140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:00.047101    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"]
Nov 01 09:35:00.054324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:00.054296    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2b9cc-2zhs7"]
Nov 01 09:35:00.073618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:00.073585    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:00.929793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:00.929759    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297f0dff-623f-43e2-bf73-39cd1c121a91" path="/var/lib/kubelet/pods/297f0dff-623f-43e2-bf73-39cd1c121a91/volumes"
Nov 01 09:35:02.051063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:02.051022    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"]
Nov 01 09:35:02.097379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:02.097338    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a3be1e-m95v2"]
Nov 01 09:35:02.930044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:02.930011    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af3f87b-9e47-4054-a99d-d4e7e113bab8" path="/var/lib/kubelet/pods/5af3f87b-9e47-4054-a99d-d4e7e113bab8/volumes"
Nov 01 09:35:04.734015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:04.733984    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:04.956841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:04.956807    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:05.102018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:05.101920    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:09.443906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:09.443873    4314 scope.go:122] "RemoveContainer" containerID="0012c1f280338f25dd634a8116de9d55130afe736d9e34625a04015bbaf14de1"
Nov 01 09:35:09.453735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:09.453708    4314 scope.go:122] "RemoveContainer" containerID="af67317600103b96dcd70ac4e5d4938c55a6bcaf8c433be8cedaaf0e830fc25c"
Nov 01 09:35:09.683131 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:09.683093    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:11.326394 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:35:11.326356    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:35:14.577857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:14.577821    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:19.223137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:19.223104    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:19.383237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:19.383189    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:19.526172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:19.526138    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:21.590788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:21.590754    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:24.176453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:24.176421    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:25.851606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.851576    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"]
Nov 01 09:35:25.852220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852204    4314 cpu_manager.go:414] "containerMap: removing container" podUID="6cb99506-a507-4313-86c6-9b47e9ed920d" containerName="payload-pre-processing"
Nov 01 09:35:25.852268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852221    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb99506-a507-4313-86c6-9b47e9ed920d" containerName="payload-pre-processing"
Nov 01 09:35:25.852268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852233    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8499201a-74b6-4f23-b97f-66a43f5df7d4" containerName="main"
Nov 01 09:35:25.852268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852238    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8499201a-74b6-4f23-b97f-66a43f5df7d4" containerName="main"
Nov 01 09:35:25.852268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852247    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" containerName="payload-processing"
Nov 01 09:35:25.852268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852253    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" containerName="payload-processing"
Nov 01 09:35:25.852458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852311    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8c67b251-a132-46ff-82eb-11985fe5814f" containerName="payload-pre-processing"
Nov 01 09:35:25.852458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852317    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c67b251-a132-46ff-82eb-11985fe5814f" containerName="payload-pre-processing"
Nov 01 09:35:25.852458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852334    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b67eb486-8b20-49ab-91e4-a621e4c4605b" containerName="main"
Nov 01 09:35:25.852458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852340    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67eb486-8b20-49ab-91e4-a621e4c4605b" containerName="main"
Nov 01 09:35:25.852458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852373    4314 cpu_manager.go:414] "containerMap: removing container" podUID="36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" containerName="payload-processing"
Nov 01 09:35:25.852458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852377    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" containerName="payload-processing"
Nov 01 09:35:25.852665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852469    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb99506-a507-4313-86c6-9b47e9ed920d" containerName="payload-pre-processing"
Nov 01 09:35:25.852665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852479    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8499201a-74b6-4f23-b97f-66a43f5df7d4" containerName="main"
Nov 01 09:35:25.852665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852486    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c493ba1b-ba21-400a-b40a-9bea6cc5f7fc" containerName="payload-processing"
Nov 01 09:35:25.852665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852497    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c67b251-a132-46ff-82eb-11985fe5814f" containerName="payload-pre-processing"
Nov 01 09:35:25.852665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852505    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67eb486-8b20-49ab-91e4-a621e4c4605b" containerName="main"
Nov 01 09:35:25.852665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.852512    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bd9eb6-4d64-4b7c-9533-e8a2ac7ebc7d" containerName="payload-processing"
Nov 01 09:35:25.864353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.857212    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:25.909351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.909317    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"]
Nov 01 09:35:25.916418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.916378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:25.916693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.916671    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:25.916809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.916786    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbjl\" (UniqueName: \"kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:25.950974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.950945    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:35:25.950974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.950967    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:35:25.951204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.951053    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:35:25.954991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:25.954962    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:35:26.018097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.018058    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.018596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.018572    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.018729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.018674    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbjl\" (UniqueName: \"kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.019146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.019124    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.020643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.020624    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.075704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.075676    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbjl\" (UniqueName: \"kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl\") pod \"maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.174460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.174374    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:26.546333 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.546305    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"]
Nov 01 09:35:26.547183 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:35:26.547159    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a068509_1359_4e05_824e_95bb1b1bb8f9.slice/crio-f0c1929ac3639f1f87984daa316164bdb8c578543932ef8df255c922e04bdf8d WatchSource:0}: Error finding container f0c1929ac3639f1f87984daa316164bdb8c578543932ef8df255c922e04bdf8d: Status 404 returned error can't find the container with id f0c1929ac3639f1f87984daa316164bdb8c578543932ef8df255c922e04bdf8d
Nov 01 09:35:26.585097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:26.585070    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf" event={"ID":"1a068509-1359-4e05-824e-95bb1b1bb8f9","Type":"ContainerStarted","Data":"f0c1929ac3639f1f87984daa316164bdb8c578543932ef8df255c922e04bdf8d"}
Nov 01 09:35:27.593876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:27.593844    4314 generic.go:356] "Generic (PLEG): container finished" podID="1a068509-1359-4e05-824e-95bb1b1bb8f9" containerID="5b86b4324f26077690c03e40ea5092851828bfdc380bff22b9bad9663a053e64" exitCode=0
Nov 01 09:35:27.594372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:27.593925    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf" event={"ID":"1a068509-1359-4e05-824e-95bb1b1bb8f9","Type":"ContainerDied","Data":"5b86b4324f26077690c03e40ea5092851828bfdc380bff22b9bad9663a053e64"}
Nov 01 09:35:28.752865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.752835    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:28.854395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.854299    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca\") pod \"1a068509-1359-4e05-824e-95bb1b1bb8f9\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") "
Nov 01 09:35:28.854395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.854381    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl\" (UniqueName: \"kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl\") pod \"1a068509-1359-4e05-824e-95bb1b1bb8f9\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") "
Nov 01 09:35:28.854669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.854601    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca\") pod \"1a068509-1359-4e05-824e-95bb1b1bb8f9\" (UID: \"1a068509-1359-4e05-824e-95bb1b1bb8f9\") "
Nov 01 09:35:28.854735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.854672    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca" pod "1a068509-1359-4e05-824e-95bb1b1bb8f9" (UID: "1a068509-1359-4e05-824e-95bb1b1bb8f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:35:28.855070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.855049    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:28.856509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.856486    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca" pod "1a068509-1359-4e05-824e-95bb1b1bb8f9" (UID: "1a068509-1359-4e05-824e-95bb1b1bb8f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:35:28.856612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.856579    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl" pod "1a068509-1359-4e05-824e-95bb1b1bb8f9" (UID: "1a068509-1359-4e05-824e-95bb1b1bb8f9"). InnerVolumeSpecName "kube-api-access-hhbjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:35:28.955867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.955837    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a068509-1359-4e05-824e-95bb1b1bb8f9-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:28.955867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:28.955863    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbjl\" (UniqueName: \"kubernetes.io/projected/1a068509-1359-4e05-824e-95bb1b1bb8f9-kube-api-access-hhbjl\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:29.485885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:29.485829    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:29.611715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:29.611687    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"
Nov 01 09:35:29.611715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:29.611697    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf" event={"ID":"1a068509-1359-4e05-824e-95bb1b1bb8f9","Type":"ContainerDied","Data":"f0c1929ac3639f1f87984daa316164bdb8c578543932ef8df255c922e04bdf8d"}
Nov 01 09:35:29.611939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:29.611746    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0c1929ac3639f1f87984daa316164bdb8c578543932ef8df255c922e04bdf8d"
Nov 01 09:35:34.570678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:34.570645    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:39.398280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.398242    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"]
Nov 01 09:35:39.398942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.398926    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1a068509-1359-4e05-824e-95bb1b1bb8f9" containerName="revoke-keys"
Nov 01 09:35:39.398988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.398944    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a068509-1359-4e05-824e-95bb1b1bb8f9" containerName="revoke-keys"
Nov 01 09:35:39.399067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.399057    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a068509-1359-4e05-824e-95bb1b1bb8f9" containerName="revoke-keys"
Nov 01 09:35:39.402917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.402899    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.437222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.437193    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:39.451462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.451435    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:35:39.451614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.451467    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:35:39.451614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.451472    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:35:39.490327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.490285    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.490571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.490512    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkxt\" (UniqueName: \"kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.490811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.490782    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.497611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.497573    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:35:39.501644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.501620    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"]
Nov 01 09:35:39.592231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.592197    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.592445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.592383    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkxt\" (UniqueName: \"kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.592598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.592580    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.593415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.593385    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.594640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.594614    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.649868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.649790    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkxt\" (UniqueName: \"kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt\") pod \"maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:39.716260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:39.716227    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:40.087615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:40.087584    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"]
Nov 01 09:35:40.088607 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:35:40.088572    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2933d0e3_e6e6_4a6e_87a4_f7b939e32257.slice/crio-489f98fe03db067500f3c70a9d8bb943a5759a865d11b4aee615c1b8dec5c201 WatchSource:0}: Error finding container 489f98fe03db067500f3c70a9d8bb943a5759a865d11b4aee615c1b8dec5c201: Status 404 returned error can't find the container with id 489f98fe03db067500f3c70a9d8bb943a5759a865d11b4aee615c1b8dec5c201
Nov 01 09:35:40.689580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:40.689440    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn" event={"ID":"2933d0e3-e6e6-4a6e-87a4-f7b939e32257","Type":"ContainerStarted","Data":"495ae6aee441d9f2524dfb4183ba963eaf21ce3e44220203c1a7fdf0dbc034f7"}
Nov 01 09:35:40.689580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:40.689511    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn" event={"ID":"2933d0e3-e6e6-4a6e-87a4-f7b939e32257","Type":"ContainerStarted","Data":"489f98fe03db067500f3c70a9d8bb943a5759a865d11b4aee615c1b8dec5c201"}
Nov 01 09:35:40.799414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:40.799341    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn" podStartSLOduration=1.799326163 podStartE2EDuration="1.799326163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:35:40.797853649 +0000 UTC m=+3813.992248205" watchObservedRunningTime="2026-11-01 09:35:40.799326163 +0000 UTC m=+3813.993720716"
Nov 01 09:35:41.326405 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:35:41.326301    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:35:41.697618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:41.697510    4314 generic.go:356] "Generic (PLEG): container finished" podID="2933d0e3-e6e6-4a6e-87a4-f7b939e32257" containerID="495ae6aee441d9f2524dfb4183ba963eaf21ce3e44220203c1a7fdf0dbc034f7" exitCode=0
Nov 01 09:35:41.697618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:41.697566    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn" event={"ID":"2933d0e3-e6e6-4a6e-87a4-f7b939e32257","Type":"ContainerDied","Data":"495ae6aee441d9f2524dfb4183ba963eaf21ce3e44220203c1a7fdf0dbc034f7"}
Nov 01 09:35:42.863035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:42.863006    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:43.016367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.016318    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:43.038917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.038891    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt\" (UniqueName: \"kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt\") pod \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") "
Nov 01 09:35:43.039062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.039011    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca\") pod \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") "
Nov 01 09:35:43.039127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.039115    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca\") pod \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\" (UID: \"2933d0e3-e6e6-4a6e-87a4-f7b939e32257\") "
Nov 01 09:35:43.039509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.039480    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca" pod "2933d0e3-e6e6-4a6e-87a4-f7b939e32257" (UID: "2933d0e3-e6e6-4a6e-87a4-f7b939e32257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:35:43.041238 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.041206    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca" pod "2933d0e3-e6e6-4a6e-87a4-f7b939e32257" (UID: "2933d0e3-e6e6-4a6e-87a4-f7b939e32257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:35:43.041343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.041327    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt" pod "2933d0e3-e6e6-4a6e-87a4-f7b939e32257" (UID: "2933d0e3-e6e6-4a6e-87a4-f7b939e32257"). InnerVolumeSpecName "kube-api-access-xwkxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:35:43.140335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.140290    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:43.140335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.140330    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:43.140335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.140341    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwkxt\" (UniqueName: \"kubernetes.io/projected/2933d0e3-e6e6-4a6e-87a4-f7b939e32257-kube-api-access-xwkxt\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:43.162683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.162654    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:43.713155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.713124    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"
Nov 01 09:35:43.713345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.713123    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn" event={"ID":"2933d0e3-e6e6-4a6e-87a4-f7b939e32257","Type":"ContainerDied","Data":"489f98fe03db067500f3c70a9d8bb943a5759a865d11b4aee615c1b8dec5c201"}
Nov 01 09:35:43.713345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:43.713248    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489f98fe03db067500f3c70a9d8bb943a5759a865d11b4aee615c1b8dec5c201"
Nov 01 09:35:44.385608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:44.385573    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:49.183249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:49.183156    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:53.924223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:53.924180    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:55.782169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.781907    4314 patch_prober.go:29] interesting pod/e2e-autores-1f3121-openshift-default-797c788959-klwt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": read tcp 10.132.0.2:58666->10.132.1.21:15021: read: connection reset by peer" start-of-body=
Nov 01 09:35:55.782169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.781971    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": read tcp 10.132.0.2:58666->10.132.1.21:15021: read: connection reset by peer"
Nov 01 09:35:55.825508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.825468    4314 generic.go:356] "Generic (PLEG): container finished" podID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerID="31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36" exitCode=0
Nov 01 09:35:55.825826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.825776    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" event={"ID":"4eb60f2c-7c8d-438b-bde4-45f9034fce60","Type":"ContainerDied","Data":"31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"}
Nov 01 09:35:55.826383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.826359    4314 scope.go:122] "RemoveContainer" containerID="31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"
Nov 01 09:35:55.832968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.832931    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:35:55.833061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.833028    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:35:55.833102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:55.833071    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:35:56.835145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:56.835106    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" event={"ID":"4eb60f2c-7c8d-438b-bde4-45f9034fce60","Type":"ContainerStarted","Data":"de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05"}
Nov 01 09:35:57.342619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:57.342582    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:35:57.342816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:57.342795    4314 patch_prober.go:29] interesting pod/e2e-autores-1f3121-openshift-default-797c788959-klwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 01 09:35:57.342860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:57.342841    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 01 09:35:58.116497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:58.116459    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:35:58.342242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:58.342207    4314 patch_prober.go:29] interesting pod/e2e-autores-1f3121-openshift-default-797c788959-klwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 01 09:35:58.342433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:58.342271    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 01 09:35:59.342290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:59.342254    4314 patch_prober.go:29] interesting pod/e2e-autores-1f3121-openshift-default-797c788959-klwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 01 09:35:59.342752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:59.342308    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 01 09:35:59.389277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:59.389238    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"]
Nov 01 09:35:59.389818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:59.389780    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy" containerID="cri-o://98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01" gracePeriod=30
Nov 01 09:35:59.846863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:35:59.846822    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:00.342423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:00.342392    4314 patch_prober.go:29] interesting pod/e2e-autores-1f3121-openshift-default-797c788959-klwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 01 09:36:00.342862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:00.342457    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 01 09:36:01.342120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:01.342078    4314 patch_prober.go:29] interesting pod/e2e-autores-1f3121-openshift-default-797c788959-klwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 01 09:36:01.342310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:01.342139    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 01 09:36:02.346741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:02.346709    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:36:02.347162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:02.347123    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:36:02.347801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:02.347780    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:36:04.681249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.681221    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:36:04.704063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.704025    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:04.785651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785619    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.785853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785667    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.785853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785700    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.785853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785739    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.786025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785860    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.786025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785940    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.786025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.785995    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.786206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786063    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:04.786206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786087    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:36:04.786206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786103    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.786206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786187    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:04.786437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786227    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:04.786437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786247    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs\") pod \"caf4b813-2a29-4707-b5fa-e72b028fa369\" (UID: \"caf4b813-2a29-4707-b5fa-e72b028fa369\") "
Nov 01 09:36:04.786570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786491    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:04.786851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786831    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.786930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786855    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.786930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786872    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.786930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786886    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf4b813-2a29-4707-b5fa-e72b028fa369-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.786930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.786900    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.788599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.788569    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:36:04.788738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.788719    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:04.788818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.788795    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:04.790130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.790108    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z" pod "caf4b813-2a29-4707-b5fa-e72b028fa369" (UID: "caf4b813-2a29-4707-b5fa-e72b028fa369"). InnerVolumeSpecName "kube-api-access-spq6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:04.858314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.858276    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:04.888196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.888163    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spq6z\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-kube-api-access-spq6z\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.888196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.888196    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.888515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.888208    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.888515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.888221    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf4b813-2a29-4707-b5fa-e72b028fa369-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:04.919309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.919227    4314 generic.go:356] "Generic (PLEG): container finished" podID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerID="98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01" exitCode=0
Nov 01 09:36:04.919309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.919284    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"
Nov 01 09:36:04.919309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.919297    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" event={"ID":"caf4b813-2a29-4707-b5fa-e72b028fa369","Type":"ContainerDied","Data":"98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01"}
Nov 01 09:36:04.919634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.919337    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5" event={"ID":"caf4b813-2a29-4707-b5fa-e72b028fa369","Type":"ContainerDied","Data":"fa2a40b88436ccee638c00dcc3ec9d10b6623d6df68d9e9033414c401ef0519a"}
Nov 01 09:36:04.919634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.919359    4314 scope.go:122] "RemoveContainer" containerID="98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01"
Nov 01 09:36:04.931558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.931507    4314 scope.go:122] "RemoveContainer" containerID="98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01"
Nov 01 09:36:04.931886 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:04.931862    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01\": container with ID starting with 98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01 not found: ID does not exist" containerID="98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01"
Nov 01 09:36:04.931990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:04.931892    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01"} err="failed to get container status \"98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01\": rpc error: code = NotFound desc = could not find container \"98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01\": container with ID starting with 98d8bb08a61eceeac5e3baeb20476d695b3d3f19487e6192121cdf6e62420a01 not found: ID does not exist"
Nov 01 09:36:05.075091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:05.075055    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"]
Nov 01 09:36:05.081391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:05.081355    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-zp4d5"]
Nov 01 09:36:06.400609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:06.400572    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:06.931932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:06.931900    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" path="/var/lib/kubelet/pods/caf4b813-2a29-4707-b5fa-e72b028fa369/volumes"
Nov 01 09:36:07.003415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:07.003381    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"]
Nov 01 09:36:07.003640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:07.003615    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy" containerID="cri-o://95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623" gracePeriod=30
Nov 01 09:36:07.008647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:07.008609    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:07.794727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:07.794690    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:09.500367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:09.500330    4314 scope.go:122] "RemoveContainer" containerID="9b78fc1c1e1e5412a1668d03a983bac1e0e315d5340432206cebf88f17cecdef"
Nov 01 09:36:09.510274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:09.510254    4314 scope.go:122] "RemoveContainer" containerID="c0cd4af34c6249d946a3d1dcdfc82b65dd159cdb76e7245e46b2a73f7bc87120"
Nov 01 09:36:09.519832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:09.519807    4314 scope.go:122] "RemoveContainer" containerID="1645fa07501a44b0c7952c6cb69df47c7f27fc12a4ca55662f867a465e17ad66"
Nov 01 09:36:09.529191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:09.529171    4314 scope.go:122] "RemoveContainer" containerID="56a6fb343dbcf3d808e0f9bd47f45edb1f10771dd18a6a51a9c44a62ebae76cf"
Nov 01 09:36:09.822128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:09.822038    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:11.329342 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:11.329305    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:36:12.281550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.281467    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:36:12.406982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.406949    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.406988    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407015    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407092    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407131    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407276    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407352    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407379    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407416    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407480    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy\") pod \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\" (UID: \"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06\") "
Nov 01 09:36:12.407969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407493    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:36:12.407969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407497    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:12.407969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407682    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:12.407969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.407765    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:12.408124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.408098    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.408160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.408123    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.408160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.408142    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.408226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.408157    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.408226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.408171    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.409467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.409435    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:36:12.409623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.409602    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "kube-api-access-j5q62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:12.409761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.409738    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:12.409857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.409841    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy" pod "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" (UID: "4d797966-8bdd-49fc-8c2e-f20b3f5ffe06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:36:12.509893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.509851    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5q62\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-kube-api-access-j5q62\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.509893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.509886    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.509893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.509897    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.510204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.509905    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:12.977734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.977701    4314 generic.go:356] "Generic (PLEG): container finished" podID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerID="95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623" exitCode=0
Nov 01 09:36:12.977923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.977752    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"
Nov 01 09:36:12.977923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.977788    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" event={"ID":"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06","Type":"ContainerDied","Data":"95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623"}
Nov 01 09:36:12.977923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.977847    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46" event={"ID":"4d797966-8bdd-49fc-8c2e-f20b3f5ffe06","Type":"ContainerDied","Data":"a4193ac24ef08cd4b04f53f8bb16d4273b23fb7010006d30b407178785c15abf"}
Nov 01 09:36:12.977923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.977875    4314 scope.go:122] "RemoveContainer" containerID="95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623"
Nov 01 09:36:12.988133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.988108    4314 scope.go:122] "RemoveContainer" containerID="c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"
Nov 01 09:36:12.997708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.997689    4314 scope.go:122] "RemoveContainer" containerID="95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623"
Nov 01 09:36:12.998006 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:12.997983    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623\": container with ID starting with 95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623 not found: ID does not exist" containerID="95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623"
Nov 01 09:36:12.998076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.998014    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623"} err="failed to get container status \"95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623\": rpc error: code = NotFound desc = could not find container \"95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623\": container with ID starting with 95b2f3a6ea79bb402323c9561157e3bf77ba2b50f768275abd0c79cadc54e623 not found: ID does not exist"
Nov 01 09:36:12.998076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.998036    4314 scope.go:122] "RemoveContainer" containerID="c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"
Nov 01 09:36:12.998312 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:12.998293    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964\": container with ID starting with c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964 not found: ID does not exist" containerID="c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"
Nov 01 09:36:12.998356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:12.998320    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964"} err="failed to get container status \"c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964\": rpc error: code = NotFound desc = could not find container \"c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964\": container with ID starting with c1c41fee5ad6468fcc31ada2c17d43b82924c18ddda9e259b558e4d17b8c0964 not found: ID does not exist"
Nov 01 09:36:13.133952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:13.133921    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"]
Nov 01 09:36:13.180999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:13.180958    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-54c25e-openshift-default-648d8b8dcf-h9x46"]
Nov 01 09:36:14.032737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:14.032705    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:14.746827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:14.746793    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"]
Nov 01 09:36:14.850280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:14.850250    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"]
Nov 01 09:36:14.929841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:14.929809    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" path="/var/lib/kubelet/pods/4d797966-8bdd-49fc-8c2e-f20b3f5ffe06/volumes"
Nov 01 09:36:15.095192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:15.095113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:15.249414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:15.249376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:19.739931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:19.739898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:19.747661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:19.747633    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb" podUID="ceffbb2d-316d-4fff-92cc-c20fdb13ad56" containerName="payload-processing" containerID="cri-o://e205666664521652a2503209edb6ed976d785029f96219553df2b84c0147eed0" gracePeriod=25
Nov 01 09:36:19.851096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:19.851056    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q" podUID="70fedd6d-c351-462f-b8b1-e3193d5fb7d9" containerName="payload-pre-processing" containerID="cri-o://adcbd9f9120b7a2562b79f43d0677028ad05015d4e968504f3aa1e010d13b838" gracePeriod=25
Nov 01 09:36:20.028587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.028526    4314 generic.go:356] "Generic (PLEG): container finished" podID="70fedd6d-c351-462f-b8b1-e3193d5fb7d9" containerID="adcbd9f9120b7a2562b79f43d0677028ad05015d4e968504f3aa1e010d13b838" exitCode=0
Nov 01 09:36:20.028776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.028587    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q" event={"ID":"70fedd6d-c351-462f-b8b1-e3193d5fb7d9","Type":"ContainerDied","Data":"adcbd9f9120b7a2562b79f43d0677028ad05015d4e968504f3aa1e010d13b838"}
Nov 01 09:36:20.030492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.030468    4314 generic.go:356] "Generic (PLEG): container finished" podID="ceffbb2d-316d-4fff-92cc-c20fdb13ad56" containerID="e205666664521652a2503209edb6ed976d785029f96219553df2b84c0147eed0" exitCode=0
Nov 01 09:36:20.030635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.030591    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb" event={"ID":"ceffbb2d-316d-4fff-92cc-c20fdb13ad56","Type":"ContainerDied","Data":"e205666664521652a2503209edb6ed976d785029f96219553df2b84c0147eed0"}
Nov 01 09:36:20.030635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.030631    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb" event={"ID":"ceffbb2d-316d-4fff-92cc-c20fdb13ad56","Type":"ContainerDied","Data":"ce34dac796ddb8bfba3d9f09680dbcdb44f8b355810544da0522662ba2d2847a"}
Nov 01 09:36:20.030722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.030646    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce34dac796ddb8bfba3d9f09680dbcdb44f8b355810544da0522662ba2d2847a"
Nov 01 09:36:20.034821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.034796    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:36:20.105882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.105838    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr\" (UniqueName: \"kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr\") pod \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") "
Nov 01 09:36:20.106106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.105992    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume\") pod \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\" (UID: \"ceffbb2d-316d-4fff-92cc-c20fdb13ad56\") "
Nov 01 09:36:20.106382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.106359    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume" pod "ceffbb2d-316d-4fff-92cc-c20fdb13ad56" (UID: "ceffbb2d-316d-4fff-92cc-c20fdb13ad56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:36:20.106609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.106592    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:20.108823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.108793    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr" pod "ceffbb2d-316d-4fff-92cc-c20fdb13ad56" (UID: "ceffbb2d-316d-4fff-92cc-c20fdb13ad56"). InnerVolumeSpecName "kube-api-access-9g9gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:20.135380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.135352    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:36:20.207444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.207403    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume\") pod \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") "
Nov 01 09:36:20.207695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.207595    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk\" (UniqueName: \"kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk\") pod \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\" (UID: \"70fedd6d-c351-462f-b8b1-e3193d5fb7d9\") "
Nov 01 09:36:20.207908 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.207883    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume" pod "70fedd6d-c351-462f-b8b1-e3193d5fb7d9" (UID: "70fedd6d-c351-462f-b8b1-e3193d5fb7d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:36:20.207980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.207959    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9gr\" (UniqueName: \"kubernetes.io/projected/ceffbb2d-316d-4fff-92cc-c20fdb13ad56-kube-api-access-9g9gr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:20.209751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.209723    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk" pod "70fedd6d-c351-462f-b8b1-e3193d5fb7d9" (UID: "70fedd6d-c351-462f-b8b1-e3193d5fb7d9"). InnerVolumeSpecName "kube-api-access-4bwzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:20.309549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.309436    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwzk\" (UniqueName: \"kubernetes.io/projected/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-kube-api-access-4bwzk\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:20.309549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.309469    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fedd6d-c351-462f-b8b1-e3193d5fb7d9-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:20.335705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:20.335659    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:21.040159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.040072    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"
Nov 01 09:36:21.040159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.040089    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q" event={"ID":"70fedd6d-c351-462f-b8b1-e3193d5fb7d9","Type":"ContainerDied","Data":"1e7e353eb8b0c44d40e36e87c0e7e7c3c2b017543f185ff64fe50c663f6338ff"}
Nov 01 09:36:21.040159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.040161    4314 scope.go:122] "RemoveContainer" containerID="adcbd9f9120b7a2562b79f43d0677028ad05015d4e968504f3aa1e010d13b838"
Nov 01 09:36:21.040699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.040359    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"
Nov 01 09:36:21.194764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.194728    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"]
Nov 01 09:36:21.241954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.241913    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8071b-69d6f4974c-2q88q"]
Nov 01 09:36:21.302020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.301907    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"]
Nov 01 09:36:21.305636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.305605    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8071b-8b676fb5f-7vldb"]
Nov 01 09:36:21.935882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:21.935847    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:22.929098 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:22.929065    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fedd6d-c351-462f-b8b1-e3193d5fb7d9" path="/var/lib/kubelet/pods/70fedd6d-c351-462f-b8b1-e3193d5fb7d9/volumes"
Nov 01 09:36:22.929492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:22.929479    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceffbb2d-316d-4fff-92cc-c20fdb13ad56" path="/var/lib/kubelet/pods/ceffbb2d-316d-4fff-92cc-c20fdb13ad56/volumes"
Nov 01 09:36:25.440341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:25.440294    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:30.196024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:30.195989    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:33.995324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:33.995287    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:34.985381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:34.985340    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:39.765851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:39.765814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:40.347756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:40.347721    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:41.324890 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:41.324852    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:36:44.977394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:44.977362    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:45.642414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:45.642373    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"]
Nov 01 09:36:45.742354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:45.742313    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"]
Nov 01 09:36:45.991382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:45.991351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:46.146066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:46.146034    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:46.975588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:46.974337    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:50.293521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:50.293481    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:50.643359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:50.643249    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5" podUID="40c3635b-2510-4128-808a-3b03c5a34e9a" containerName="payload-processing" containerID="cri-o://e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584" gracePeriod=25
Nov 01 09:36:50.742648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:50.742593    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh" podUID="ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" containerName="payload-pre-processing" containerID="cri-o://e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26" gracePeriod=25
Nov 01 09:36:50.912858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:50.912497    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:36:51.016828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.016799    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume\") pod \"40c3635b-2510-4128-808a-3b03c5a34e9a\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") "
Nov 01 09:36:51.016989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.016943    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6\" (UniqueName: \"kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6\") pod \"40c3635b-2510-4128-808a-3b03c5a34e9a\" (UID: \"40c3635b-2510-4128-808a-3b03c5a34e9a\") "
Nov 01 09:36:51.017242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.017215    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume" pod "40c3635b-2510-4128-808a-3b03c5a34e9a" (UID: "40c3635b-2510-4128-808a-3b03c5a34e9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:36:51.017547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.017515    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c3635b-2510-4128-808a-3b03c5a34e9a-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:51.019294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.019271    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6" pod "40c3635b-2510-4128-808a-3b03c5a34e9a" (UID: "40c3635b-2510-4128-808a-3b03c5a34e9a"). InnerVolumeSpecName "kube-api-access-8kgm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:51.021099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.021075    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:36:51.118614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.118571    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2\" (UniqueName: \"kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2\") pod \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") "
Nov 01 09:36:51.118783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.118754    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume\") pod \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\" (UID: \"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7\") "
Nov 01 09:36:51.119152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.119122    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume" pod "ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" (UID: "ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:36:51.119329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.119315    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgm6\" (UniqueName: \"kubernetes.io/projected/40c3635b-2510-4128-808a-3b03c5a34e9a-kube-api-access-8kgm6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:51.119376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.119332    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:51.120689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.120661    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2" pod "ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" (UID: "ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7"). InnerVolumeSpecName "kube-api-access-sdxx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:36:51.220679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.220597    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxx2\" (UniqueName: \"kubernetes.io/projected/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7-kube-api-access-sdxx2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:36:51.251910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.251879    4314 generic.go:356] "Generic (PLEG): container finished" podID="ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" containerID="e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26" exitCode=0
Nov 01 09:36:51.252082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.251924    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"
Nov 01 09:36:51.252082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.251960    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh" event={"ID":"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7","Type":"ContainerDied","Data":"e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26"}
Nov 01 09:36:51.252082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.252012    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh" event={"ID":"ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7","Type":"ContainerDied","Data":"0d93d4c7da737033708e5a9ad56c7ab8d16c666c59258bade97266ccedbb2af9"}
Nov 01 09:36:51.252082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.252042    4314 scope.go:122] "RemoveContainer" containerID="e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26"
Nov 01 09:36:51.253365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.253345    4314 generic.go:356] "Generic (PLEG): container finished" podID="40c3635b-2510-4128-808a-3b03c5a34e9a" containerID="e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584" exitCode=0
Nov 01 09:36:51.253469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.253380    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"
Nov 01 09:36:51.253469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.253430    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5" event={"ID":"40c3635b-2510-4128-808a-3b03c5a34e9a","Type":"ContainerDied","Data":"e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584"}
Nov 01 09:36:51.253649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.253482    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5" event={"ID":"40c3635b-2510-4128-808a-3b03c5a34e9a","Type":"ContainerDied","Data":"9475805188d5a12950271002833533aa2f89e61fe3ded730e54268e631dcc32c"}
Nov 01 09:36:51.263862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.263841    4314 scope.go:122] "RemoveContainer" containerID="e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26"
Nov 01 09:36:51.264223 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:51.264192    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26\": container with ID starting with e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26 not found: ID does not exist" containerID="e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26"
Nov 01 09:36:51.264335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.264234    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26"} err="failed to get container status \"e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26\": rpc error: code = NotFound desc = could not find container \"e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26\": container with ID starting with e57a62fec1a21bec34050889d12304dac6ca7b9041a8616b16e028b4a52c8d26 not found: ID does not exist"
Nov 01 09:36:51.264335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.264259    4314 scope.go:122] "RemoveContainer" containerID="e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584"
Nov 01 09:36:51.275126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.275100    4314 scope.go:122] "RemoveContainer" containerID="e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584"
Nov 01 09:36:51.275436 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:36:51.275414    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584\": container with ID starting with e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584 not found: ID does not exist" containerID="e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584"
Nov 01 09:36:51.275500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.275445    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584"} err="failed to get container status \"e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584\": rpc error: code = NotFound desc = could not find container \"e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584\": container with ID starting with e85bbd79c4b6c6d16137749380344481ce14b32f3db65296be2d13aecaff0584 not found: ID does not exist"
Nov 01 09:36:51.406956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.406920    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"]
Nov 01 09:36:51.411284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.411252    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-49e170-6698d5d4dc-5tjvh"]
Nov 01 09:36:51.516171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.516106    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"]
Nov 01 09:36:51.521312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:51.521272    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-49e170-6b5dfcfcf9-xdzf5"]
Nov 01 09:36:52.928985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:52.928954    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c3635b-2510-4128-808a-3b03c5a34e9a" path="/var/lib/kubelet/pods/40c3635b-2510-4128-808a-3b03c5a34e9a/volumes"
Nov 01 09:36:52.929386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:52.929363    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" path="/var/lib/kubelet/pods/ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7/volumes"
Nov 01 09:36:53.081278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:53.081243    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:53.740270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:53.740236    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:53.936566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:53.936497    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:55.408205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:55.408174    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:55.557449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:55.557419    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:58.676457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:58.676414    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:36:59.749902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:36:59.749864    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:01.535648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:01.535606    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:01.684110 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:01.684078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:01.745838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:01.745799    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"]
Nov 01 09:37:01.791277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:01.791200    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-25d9fe-wr4vf"]
Nov 01 09:37:01.896109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:01.896072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:02.007812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:02.007773    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"]
Nov 01 09:37:02.008111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:02.008081    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" containerID="cri-o://cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b" gracePeriod=30
Nov 01 09:37:02.047400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:02.047311    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:02.572802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:02.572755    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:37:02.573235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:02.572852    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:37:02.929123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:02.929046    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a068509-1359-4e05-824e-95bb1b1bb8f9" path="/var/lib/kubelet/pods/1a068509-1359-4e05-824e-95bb1b1bb8f9/volumes"
Nov 01 09:37:05.605489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:05.605447    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:05.756597 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:05.756563    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:06.919545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:06.919499    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:07.286095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.286065    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:37:07.334860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.334828    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.334896    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.334938    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.334979    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335001    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335040    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335093    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335126    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335187    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token\") pod \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\" (UID: \"22fe3e44-434b-47d5-88e6-de80b6d2e0cc\") "
Nov 01 09:37:07.335491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335323    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:37:07.335491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335353    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:37:07.335491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335377    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:07.335491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335389    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:37:07.335491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335446    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:37:07.335823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335690    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.335823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335713    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.335823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335727    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.335823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335741    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.335823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.335754    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.337238 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.337216    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:37:07.337405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.337391    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "kube-api-access-8bz74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:07.337461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.337441    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:07.337583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.337563    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy" pod "22fe3e44-434b-47d5-88e6-de80b6d2e0cc" (UID: "22fe3e44-434b-47d5-88e6-de80b6d2e0cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:37:07.365125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.365093    4314 generic.go:356] "Generic (PLEG): container finished" podID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerID="cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b" exitCode=0
Nov 01 09:37:07.365271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.365134    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" event={"ID":"22fe3e44-434b-47d5-88e6-de80b6d2e0cc","Type":"ContainerDied","Data":"cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b"}
Nov 01 09:37:07.365271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.365154    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"
Nov 01 09:37:07.365271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.365202    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj" event={"ID":"22fe3e44-434b-47d5-88e6-de80b6d2e0cc","Type":"ContainerDied","Data":"41549e1a31f032b95fb7e32926419f3e66e51c87eb5fb1363faaa2843f16dcd1"}
Nov 01 09:37:07.365271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.365235    4314 scope.go:122] "RemoveContainer" containerID="cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b"
Nov 01 09:37:07.376158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.376132    4314 scope.go:122] "RemoveContainer" containerID="bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"
Nov 01 09:37:07.385511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.385491    4314 scope.go:122] "RemoveContainer" containerID="cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b"
Nov 01 09:37:07.385820 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:37:07.385800    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b\": container with ID starting with cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b not found: ID does not exist" containerID="cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b"
Nov 01 09:37:07.385896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.385830    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b"} err="failed to get container status \"cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b\": rpc error: code = NotFound desc = could not find container \"cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b\": container with ID starting with cfc63863791b9e26501203a5cb1394922e78c3f3e3a9628d23932ead724d7b4b not found: ID does not exist"
Nov 01 09:37:07.385896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.385848    4314 scope.go:122] "RemoveContainer" containerID="bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"
Nov 01 09:37:07.386080 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:37:07.386065    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da\": container with ID starting with bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da not found: ID does not exist" containerID="bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"
Nov 01 09:37:07.386126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.386084    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da"} err="failed to get container status \"bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da\": rpc error: code = NotFound desc = could not find container \"bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da\": container with ID starting with bd5a44854477e356c695c011ced386a0141c3c8be8799c808da50dadb0a4d9da not found: ID does not exist"
Nov 01 09:37:07.437237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.437147    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.437237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.437189    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bz74\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-kube-api-access-8bz74\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.437237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.437205    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.437237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.437219    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fe3e44-434b-47d5-88e6-de80b6d2e0cc-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:07.551177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.551144    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"]
Nov 01 09:37:07.555427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:07.555396    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-25d9fe-openshift-default-56786b8445-hjlzj"]
Nov 01 09:37:08.148370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.148334    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:08.654278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654230    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55767669c-jc4kj"]
Nov 01 09:37:08.654928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654903    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" containerName="payload-pre-processing"
Nov 01 09:37:08.654928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654924    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" containerName="payload-pre-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654940    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ceffbb2d-316d-4fff-92cc-c20fdb13ad56" containerName="payload-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654945    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceffbb2d-316d-4fff-92cc-c20fdb13ad56" containerName="payload-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654953    4314 cpu_manager.go:414] "containerMap: removing container" podUID="70fedd6d-c351-462f-b8b1-e3193d5fb7d9" containerName="payload-pre-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654958    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fedd6d-c351-462f-b8b1-e3193d5fb7d9" containerName="payload-pre-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654971    4314 cpu_manager.go:414] "containerMap: removing container" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654976    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654986    4314 cpu_manager.go:414] "containerMap: removing container" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.654991    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655006    4314 cpu_manager.go:414] "containerMap: removing container" podUID="40c3635b-2510-4128-808a-3b03c5a34e9a" containerName="payload-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655010    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c3635b-2510-4128-808a-3b03c5a34e9a" containerName="payload-processing"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655030    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655035    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655042    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2933d0e3-e6e6-4a6e-87a4-f7b939e32257" containerName="revoke-keys"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655046    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2933d0e3-e6e6-4a6e-87a4-f7b939e32257" containerName="revoke-keys"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655057    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy"
Nov 01 09:37:08.655095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655061    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655152    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655163    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc3a2ec-8f92-4ac4-ad2c-3134008a0ea7" containerName="payload-pre-processing"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655172    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceffbb2d-316d-4fff-92cc-c20fdb13ad56" containerName="payload-processing"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655179    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fedd6d-c351-462f-b8b1-e3193d5fb7d9" containerName="payload-pre-processing"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655184    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655190    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf4b813-2a29-4707-b5fa-e72b028fa369" containerName="istio-proxy"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655197    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c3635b-2510-4128-808a-3b03c5a34e9a" containerName="payload-processing"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655205    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2933d0e3-e6e6-4a6e-87a4-f7b939e32257" containerName="revoke-keys"
Nov 01 09:37:08.655630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.655209    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy"
Nov 01 09:37:08.660096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.660077    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.709404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.709370    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 09:37:08.712475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.712446    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55767669c-jc4kj"]
Nov 01 09:37:08.752719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.752676    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.752938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.752820    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.752938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.752927    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.753066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.753012    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2v7\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.754414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.754392    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 09:37:08.754651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.754445    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 09:37:08.754651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.754453    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 09:37:08.754810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.754456    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-47f306d7be56\""
Nov 01 09:37:08.754810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.754520    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-p8pf8\""
Nov 01 09:37:08.853828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.853795    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.854015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.853872    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.854015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.853907    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2v7\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.854015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.853994    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.854578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.854555    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.854699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.854664    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.856332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.856314    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.914345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.914275    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2v7\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7\") pod \"payload-processing-external-model-55767669c-jc4kj\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") " pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:08.928989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.928951    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" path="/var/lib/kubelet/pods/22fe3e44-434b-47d5-88e6-de80b6d2e0cc/volumes"
Nov 01 09:37:08.974396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:08.974362    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:09.397119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:09.397091    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55767669c-jc4kj"]
Nov 01 09:37:09.398014 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:37:09.397990    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b55f1e_50ce_4af0_a02d_9874d34e4bf6.slice/crio-8fe255e2408f7396ba882b5476efbf1e40f0035788ae2e60961821a0c7ebd796 WatchSource:0}: Error finding container 8fe255e2408f7396ba882b5476efbf1e40f0035788ae2e60961821a0c7ebd796: Status 404 returned error can't find the container with id 8fe255e2408f7396ba882b5476efbf1e40f0035788ae2e60961821a0c7ebd796
Nov 01 09:37:10.389122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:10.389080    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj" event={"ID":"43b55f1e-50ce-4af0-a02d-9874d34e4bf6","Type":"ContainerStarted","Data":"4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27"}
Nov 01 09:37:10.389400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:10.389135    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj" event={"ID":"43b55f1e-50ce-4af0-a02d-9874d34e4bf6","Type":"ContainerStarted","Data":"8fe255e2408f7396ba882b5476efbf1e40f0035788ae2e60961821a0c7ebd796"}
Nov 01 09:37:10.389400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:10.389185    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:10.528862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:10.528804    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj" podStartSLOduration=2.5287909170000002 podStartE2EDuration="2.528790917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:37:10.527021139 +0000 UTC m=+3903.721415697" watchObservedRunningTime="2026-11-01 09:37:10.528790917 +0000 UTC m=+3903.723185502"
Nov 01 09:37:11.327600 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:37:11.327563    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:37:11.870825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:11.870786    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:16.785223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:16.785188    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:20.953937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:20.953857    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:21.403626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:21.403594    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:22.280054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:22.280021    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:22.878314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:22.878274    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"]
Nov 01 09:37:22.929409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:22.929374    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e3728d-n5tvn"]
Nov 01 09:37:23.951976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:23.951945    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:24.929714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:24.929678    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2933d0e3-e6e6-4a6e-87a4-f7b939e32257" path="/var/lib/kubelet/pods/2933d0e3-e6e6-4a6e-87a4-f7b939e32257/volumes"
Nov 01 09:37:25.929077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:25.929042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:26.101323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:26.101291    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:26.265651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:26.265616    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:26.434572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:26.428070    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:27.910675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:27.910634    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:30.143848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.143810    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"]
Nov 01 09:37:30.144488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.144467    4314 cpu_manager.go:414] "containerMap: removing container" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy"
Nov 01 09:37:30.144552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.144489    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fe3e44-434b-47d5-88e6-de80b6d2e0cc" containerName="istio-proxy"
Nov 01 09:37:30.144622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.144612    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d797966-8bdd-49fc-8c2e-f20b3f5ffe06" containerName="istio-proxy"
Nov 01 09:37:30.148227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.148209    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.199205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.199171    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0297be67dfdc\""
Nov 01 09:37:30.248746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.248710    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"]
Nov 01 09:37:30.327173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.327134    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5gb\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.327173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.327181    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.327477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.327312    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.327526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.327487    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.429137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.429038    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.429335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.429159    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.429335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.429323    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.429473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.429393    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5gb\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.430056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.430026    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.430191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.430026    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.431886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.431859    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.487003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.486973    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5gb\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb\") pod \"payload-processing-external-model-fd5d559f7-gmknr\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") " pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:30.762042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:30.762009    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:31.163852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:31.152763    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"]
Nov 01 09:37:31.537253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:31.537217    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr" event={"ID":"8c4063bd-75d1-44e0-8885-1dab686a9f29","Type":"ContainerStarted","Data":"03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e"}
Nov 01 09:37:31.537478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:31.537280    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr" event={"ID":"8c4063bd-75d1-44e0-8885-1dab686a9f29","Type":"ContainerStarted","Data":"45af8261a5d1952d938563722f4a108b2d832c3bd952750eadd591c102a71ee4"}
Nov 01 09:37:31.537478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:31.537435    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:31.650769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:31.650718    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr" podStartSLOduration=1.6507045219999998 podStartE2EDuration="1.650704522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:37:31.649047262 +0000 UTC m=+3924.843441828" watchObservedRunningTime="2026-11-01 09:37:31.650704522 +0000 UTC m=+3924.845099077"
Nov 01 09:37:32.779129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:32.779091    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:37.636115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:37.636072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:38.500193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.500156    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"]
Nov 01 09:37:38.504525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.504504    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.527998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.527969    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.528104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.528014    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmtm\" (UniqueName: \"kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.528188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.528137    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.553070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.553040    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:37:38.599169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.599142    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:37:38.602863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.602838    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"]
Nov 01 09:37:38.629072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.629046    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.629217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.629102    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmtm\" (UniqueName: \"kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.629217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.629175    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.629823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.629804    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.630171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.630145    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:37:38.630259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.630243    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:37:38.641505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.641484    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.686019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.685996    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmtm\" (UniqueName: \"kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:38.818732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:38.818656    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:39.198811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:39.198781    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"]
Nov 01 09:37:39.200206 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:37:39.200182    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99277be2_353a_4256_a7cd_6bc5f35ac62c.slice/crio-0e39e22fab2da3dde5feb5e8d33092362860ffe18b3b023733c9b104ed709936 WatchSource:0}: Error finding container 0e39e22fab2da3dde5feb5e8d33092362860ffe18b3b023733c9b104ed709936: Status 404 returned error can't find the container with id 0e39e22fab2da3dde5feb5e8d33092362860ffe18b3b023733c9b104ed709936
Nov 01 09:37:39.590340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:39.590302    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq" event={"ID":"99277be2-353a-4256-a7cd-6bc5f35ac62c","Type":"ContainerStarted","Data":"54b59ef8bd0df9c8c1a5b01d1429141dd0db924f66a213633d56ed848e4bcb0f"}
Nov 01 09:37:39.590509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:39.590365    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq" event={"ID":"99277be2-353a-4256-a7cd-6bc5f35ac62c","Type":"ContainerStarted","Data":"0e39e22fab2da3dde5feb5e8d33092362860ffe18b3b023733c9b104ed709936"}
Nov 01 09:37:39.699923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:39.699868    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq" podStartSLOduration=1.6998548690000002 podStartE2EDuration="1.699854869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:37:39.6974015 +0000 UTC m=+3932.891796095" watchObservedRunningTime="2026-11-01 09:37:39.699854869 +0000 UTC m=+3932.894249423"
Nov 01 09:37:40.598936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:40.598906    4314 generic.go:356] "Generic (PLEG): container finished" podID="99277be2-353a-4256-a7cd-6bc5f35ac62c" containerID="54b59ef8bd0df9c8c1a5b01d1429141dd0db924f66a213633d56ed848e4bcb0f" exitCode=0
Nov 01 09:37:40.599119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:40.598957    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq" event={"ID":"99277be2-353a-4256-a7cd-6bc5f35ac62c","Type":"ContainerDied","Data":"54b59ef8bd0df9c8c1a5b01d1429141dd0db924f66a213633d56ed848e4bcb0f"}
Nov 01 09:37:41.323697 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:37:41.323662    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:37:41.684134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.684101    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"]
Nov 01 09:37:41.688287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.688264    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.742695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.742664    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"]
Nov 01 09:37:41.756932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.756908    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:41.769016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.768993    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.769163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.769032    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.769310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.769290    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mkr\" (UniqueName: \"kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.870238 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870157    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca\") pod \"99277be2-353a-4256-a7cd-6bc5f35ac62c\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") "
Nov 01 09:37:41.870390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870317    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca\") pod \"99277be2-353a-4256-a7cd-6bc5f35ac62c\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") "
Nov 01 09:37:41.870390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870350    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm\" (UniqueName: \"kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm\") pod \"99277be2-353a-4256-a7cd-6bc5f35ac62c\" (UID: \"99277be2-353a-4256-a7cd-6bc5f35ac62c\") "
Nov 01 09:37:41.870576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870556    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mkr\" (UniqueName: \"kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.870660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870642    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.870717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870684    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca" pod "99277be2-353a-4256-a7cd-6bc5f35ac62c" (UID: "99277be2-353a-4256-a7cd-6bc5f35ac62c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:41.870717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870690    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.871016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.870994    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:41.871327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.871305    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.872460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.872432    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca" pod "99277be2-353a-4256-a7cd-6bc5f35ac62c" (UID: "99277be2-353a-4256-a7cd-6bc5f35ac62c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:41.872593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.872515    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm" pod "99277be2-353a-4256-a7cd-6bc5f35ac62c" (UID: "99277be2-353a-4256-a7cd-6bc5f35ac62c"). InnerVolumeSpecName "kube-api-access-7mmtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:41.872909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.872890    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.928925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.928900    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mkr\" (UniqueName: \"kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr\") pod \"maas-api-revoke-sub-d08a82c5-xs8z2\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:41.972190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.972148    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmtm\" (UniqueName: \"kubernetes.io/projected/99277be2-353a-4256-a7cd-6bc5f35ac62c-kube-api-access-7mmtm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:41.972190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:41.972183    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99277be2-353a-4256-a7cd-6bc5f35ac62c-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:42.006133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.006106    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:42.374878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.374850    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"]
Nov 01 09:37:42.375619 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:37:42.375590    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e28ba4_97f9_454a_8f7b_cfb0d09abbeb.slice/crio-fc5fc7f888700f8d8dc44f62149724e451e9326078ee372f973ab87f8cfaae74 WatchSource:0}: Error finding container fc5fc7f888700f8d8dc44f62149724e451e9326078ee372f973ab87f8cfaae74: Status 404 returned error can't find the container with id fc5fc7f888700f8d8dc44f62149724e451e9326078ee372f973ab87f8cfaae74
Nov 01 09:37:42.546381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.546350    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:37:42.614149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.614120    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"
Nov 01 09:37:42.614149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.614138    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq" event={"ID":"99277be2-353a-4256-a7cd-6bc5f35ac62c","Type":"ContainerDied","Data":"0e39e22fab2da3dde5feb5e8d33092362860ffe18b3b023733c9b104ed709936"}
Nov 01 09:37:42.614422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.614181    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e39e22fab2da3dde5feb5e8d33092362860ffe18b3b023733c9b104ed709936"
Nov 01 09:37:42.615579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.615553    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2" event={"ID":"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb","Type":"ContainerStarted","Data":"fc5fc7f888700f8d8dc44f62149724e451e9326078ee372f973ab87f8cfaae74"}
Nov 01 09:37:42.684696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.684666    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55767669c-jc4kj"]
Nov 01 09:37:42.689156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:42.689133    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:43.302039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.302007    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"]
Nov 01 09:37:43.302640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.302626    4314 cpu_manager.go:414] "containerMap: removing container" podUID="99277be2-353a-4256-a7cd-6bc5f35ac62c" containerName="revoke-keys"
Nov 01 09:37:43.302689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.302642    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99277be2-353a-4256-a7cd-6bc5f35ac62c" containerName="revoke-keys"
Nov 01 09:37:43.302750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.302741    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="99277be2-353a-4256-a7cd-6bc5f35ac62c" containerName="revoke-keys"
Nov 01 09:37:43.306095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.306074    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.355056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.355022    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\""
Nov 01 09:37:43.390397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390361    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390410    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390522    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390625    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390721    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390742    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390849    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.390883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390872    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.391208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.390924    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpdp\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.404384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.404357    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"]
Nov 01 09:37:43.492447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492411    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492459    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492498    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpdp\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492593    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492626    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492863    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.492952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492942    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.493125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492956    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.493125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492979    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.493125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.492998    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.493125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.493089    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.493365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.493289    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.493479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.493457    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.495145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.495121    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.495341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.495321    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.550296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.550255    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.550448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.550381    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpdp\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qj28m\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:43.624205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.624170    4314 generic.go:356] "Generic (PLEG): container finished" podID="b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" containerID="270ad5c85032e7defda08e1c24dc24c48c63bf48403237d680922d5202cd498b" exitCode=0
Nov 01 09:37:43.624391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.624253    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2" event={"ID":"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb","Type":"ContainerDied","Data":"270ad5c85032e7defda08e1c24dc24c48c63bf48403237d680922d5202cd498b"}
Nov 01 09:37:43.633363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:43.633336    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:44.057581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.057551    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"]
Nov 01 09:37:44.059491 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:37:44.059464    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7be1b21_71cd_4628_a60f_af4599c7ef91.slice/crio-7ed11b52e560e3f3aec938729e7e5c4ff7112af990001d0c306e5d4bf1eb6416 WatchSource:0}: Error finding container 7ed11b52e560e3f3aec938729e7e5c4ff7112af990001d0c306e5d4bf1eb6416: Status 404 returned error can't find the container with id 7ed11b52e560e3f3aec938729e7e5c4ff7112af990001d0c306e5d4bf1eb6416
Nov 01 09:37:44.062489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.062453    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:37:44.062586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.062525    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:37:44.062586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.062577    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:37:44.637750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.637711    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" event={"ID":"a7be1b21-71cd-4628-a60f-af4599c7ef91","Type":"ContainerStarted","Data":"b7e15eeed0a0e40b87276e76718c3384c1cd917c84282175dfb0b426808c6240"}
Nov 01 09:37:44.638210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.637801    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" event={"ID":"a7be1b21-71cd-4628-a60f-af4599c7ef91","Type":"ContainerStarted","Data":"7ed11b52e560e3f3aec938729e7e5c4ff7112af990001d0c306e5d4bf1eb6416"}
Nov 01 09:37:44.753208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.753152    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podStartSLOduration=1.7531365700000001 podStartE2EDuration="1.75313657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:37:44.749893407 +0000 UTC m=+3937.944287995" watchObservedRunningTime="2026-11-01 09:37:44.75313657 +0000 UTC m=+3937.947531124"
Nov 01 09:37:44.794504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.794478    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:44.912294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.912210    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca\") pod \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") "
Nov 01 09:37:44.912294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.912279    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca\") pod \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") "
Nov 01 09:37:44.912612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.912468    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr\" (UniqueName: \"kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr\") pod \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\" (UID: \"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb\") "
Nov 01 09:37:44.912704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.912669    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca" pod "b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" (UID: "b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:44.913085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.913053    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:44.915257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.915218    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr" pod "b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" (UID: "b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb"). InnerVolumeSpecName "kube-api-access-z6mkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:44.942751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:44.939696    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca" pod "b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" (UID: "b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:45.014838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.014804    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mkr\" (UniqueName: \"kubernetes.io/projected/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-kube-api-access-z6mkr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:45.014838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.014833    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:45.633970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.633930    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:45.635508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.635484    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:45.635630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.635557    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:45.646073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.646044    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"
Nov 01 09:37:45.646073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.646064    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2" event={"ID":"b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb","Type":"ContainerDied","Data":"fc5fc7f888700f8d8dc44f62149724e451e9326078ee372f973ab87f8cfaae74"}
Nov 01 09:37:45.646656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:45.646113    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5fc7f888700f8d8dc44f62149724e451e9326078ee372f973ab87f8cfaae74"
Nov 01 09:37:46.633675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:46.633644    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:46.633900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:46.633704    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:46.901209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:46.901121    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:47.634158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:47.634116    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:47.634389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:47.634175    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:47.687719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:47.687674    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj" podUID="43b55f1e-50ce-4af0-a02d-9874d34e4bf6" containerName="payload-processing" containerID="cri-o://4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27" gracePeriod=25
Nov 01 09:37:48.013089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.013065    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:48.053937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.053908    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume\") pod \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") "
Nov 01 09:37:48.054083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.054028    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials\") pod \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") "
Nov 01 09:37:48.054083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.054054    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume\") pod \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") "
Nov 01 09:37:48.054174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.054116    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7\") pod \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\" (UID: \"43b55f1e-50ce-4af0-a02d-9874d34e4bf6\") "
Nov 01 09:37:48.054268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.054243    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume" pod "43b55f1e-50ce-4af0-a02d-9874d34e4bf6" (UID: "43b55f1e-50ce-4af0-a02d-9874d34e4bf6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:48.054549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.054515    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume" pod "43b55f1e-50ce-4af0-a02d-9874d34e4bf6" (UID: "43b55f1e-50ce-4af0-a02d-9874d34e4bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:48.054667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.054651    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:48.056275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.056254    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7" pod "43b55f1e-50ce-4af0-a02d-9874d34e4bf6" (UID: "43b55f1e-50ce-4af0-a02d-9874d34e4bf6"). InnerVolumeSpecName "kube-api-access-fg2v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:48.056372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.056283    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials" pod "43b55f1e-50ce-4af0-a02d-9874d34e4bf6" (UID: "43b55f1e-50ce-4af0-a02d-9874d34e4bf6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:48.156055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.155977    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:48.156055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.156003    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:48.156055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.156016    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2v7\" (UniqueName: \"kubernetes.io/projected/43b55f1e-50ce-4af0-a02d-9874d34e4bf6-kube-api-access-fg2v7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:48.619189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.619158    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:48.634209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.634184    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:48.634355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.634262    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:48.700734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.700697    4314 generic.go:356] "Generic (PLEG): container finished" podID="43b55f1e-50ce-4af0-a02d-9874d34e4bf6" containerID="4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27" exitCode=0
Nov 01 09:37:48.700928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.700750    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj"
Nov 01 09:37:48.700928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.700785    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj" event={"ID":"43b55f1e-50ce-4af0-a02d-9874d34e4bf6","Type":"ContainerDied","Data":"4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27"}
Nov 01 09:37:48.700928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.700846    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55767669c-jc4kj" event={"ID":"43b55f1e-50ce-4af0-a02d-9874d34e4bf6","Type":"ContainerDied","Data":"8fe255e2408f7396ba882b5476efbf1e40f0035788ae2e60961821a0c7ebd796"}
Nov 01 09:37:48.700928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.700876    4314 scope.go:122] "RemoveContainer" containerID="4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27"
Nov 01 09:37:48.711677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.711654    4314 scope.go:122] "RemoveContainer" containerID="4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27"
Nov 01 09:37:48.711946 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:37:48.711925    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27\": container with ID starting with 4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27 not found: ID does not exist" containerID="4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27"
Nov 01 09:37:48.711999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.711955    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27"} err="failed to get container status \"4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27\": rpc error: code = NotFound desc = could not find container \"4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27\": container with ID starting with 4ad6feeb9fca3011ee72494f93b3c40c6ad206f251365c9b5fac7f50b20c7c27 not found: ID does not exist"
Nov 01 09:37:48.856498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.856459    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55767669c-jc4kj"]
Nov 01 09:37:48.882685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.881031    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55767669c-jc4kj"]
Nov 01 09:37:48.929545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:48.929497    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b55f1e-50ce-4af0-a02d-9874d34e4bf6" path="/var/lib/kubelet/pods/43b55f1e-50ce-4af0-a02d-9874d34e4bf6/volumes"
Nov 01 09:37:49.634600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:49.634565    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:49.635122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:49.634619    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:50.633952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:50.633902    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:50.634156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:50.633981    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:51.634685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:51.634650    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:51.635200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:51.634708    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:51.796304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:51.796268    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:51.939462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:51.939373    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:52.634205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:52.634165    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qj28m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 01 09:37:52.634433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:52.634236    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 01 09:37:53.427965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:53.427931    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:53.856718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:53.856686    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:53.864570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:53.858469    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:53.865934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:53.865903    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:37:53.956582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:53.956553    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:54.107615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.107062    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:54.376866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.376782    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:54.532896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.532866    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:54.884562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.884507    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"]
Nov 01 09:37:54.885094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.885080    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" containerName="revoke-keys"
Nov 01 09:37:54.885144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.885096    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" containerName="revoke-keys"
Nov 01 09:37:54.885144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.885109    4314 cpu_manager.go:414] "containerMap: removing container" podUID="43b55f1e-50ce-4af0-a02d-9874d34e4bf6" containerName="payload-processing"
Nov 01 09:37:54.885144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.885115    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b55f1e-50ce-4af0-a02d-9874d34e4bf6" containerName="payload-processing"
Nov 01 09:37:54.885241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.885230    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" containerName="revoke-keys"
Nov 01 09:37:54.885241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.885237    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b55f1e-50ce-4af0-a02d-9874d34e4bf6" containerName="payload-processing"
Nov 01 09:37:54.889912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.889893    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:54.943337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.943305    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"]
Nov 01 09:37:54.980645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.980604    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:54.980645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.980645    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:54.980874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.980736    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:54.980922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:54.980871    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.082197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.082161    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.082197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.082198    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.082483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.082232    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.082483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.082338    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.083004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.082978    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.083135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.083115    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.084672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.084651    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.138970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.138902    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz\") pod \"payload-processing-external-model-85d4446d68-blrnt\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.203304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.203268    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.638070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.638041    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"]
Nov 01 09:37:55.639344 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:37:55.639304    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d2a68_cc0c_4dd4_9983_67f2763a0e6c.slice/crio-3a33f0d0b1bdefee7a6e1145701a9089164d7fee2a0bfc5526b374356d08a36c WatchSource:0}: Error finding container 3a33f0d0b1bdefee7a6e1145701a9089164d7fee2a0bfc5526b374356d08a36c: Status 404 returned error can't find the container with id 3a33f0d0b1bdefee7a6e1145701a9089164d7fee2a0bfc5526b374356d08a36c
Nov 01 09:37:55.802865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.802827    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt" event={"ID":"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c","Type":"ContainerStarted","Data":"b2974a2069b927104976be87243f6e1210d8275db30f7588a218229554ee6509"}
Nov 01 09:37:55.803073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.802893    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt" event={"ID":"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c","Type":"ContainerStarted","Data":"3a33f0d0b1bdefee7a6e1145701a9089164d7fee2a0bfc5526b374356d08a36c"}
Nov 01 09:37:55.803167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.803146    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:37:55.913489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.913391    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt" podStartSLOduration=1.913376593 podStartE2EDuration="1.913376593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:37:55.912130997 +0000 UTC m=+3949.106525555" watchObservedRunningTime="2026-11-01 09:37:55.913376593 +0000 UTC m=+3949.107771175"
Nov 01 09:37:55.968736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:55.968697    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:56.403419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:56.403383    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:56.431842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:56.431811    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"]
Nov 01 09:37:56.432071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:56.432031    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy" containerID="cri-o://dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7" gracePeriod=30
Nov 01 09:37:56.551298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:56.551267    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:57.221917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:57.221883    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:58.718799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:58.718758    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:37:59.711180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.711146    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"]
Nov 01 09:37:59.754757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.754719    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"]
Nov 01 09:37:59.758880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.758863    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.807338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.807308    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-31239b0f3c57\""
Nov 01 09:37:59.815858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.815828    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"]
Nov 01 09:37:59.845455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.845419    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbcn\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.845669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.845489    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.845728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.845679    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.845898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.845874    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.946634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.946586    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.946845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.946691    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.946845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.946836    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.946987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.946861    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbcn\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.947340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.947315    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.947603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.947584    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:37:59.949325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:37:59.949304    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:00.004101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.004065    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbcn\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn\") pod \"payload-processing-external-model-7fc854dd88-f4rbq\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:00.075545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.075508    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:00.448510 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.448474    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"]
Nov 01 09:38:00.449401 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:00.449371    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a81e99_fb8b_4771_a602_1ad94c4420c7.slice/crio-104f6dca1566f0d88222ad28e6d0f27308d66191a6ccc4e9a09a2b3fac2013ba WatchSource:0}: Error finding container 104f6dca1566f0d88222ad28e6d0f27308d66191a6ccc4e9a09a2b3fac2013ba: Status 404 returned error can't find the container with id 104f6dca1566f0d88222ad28e6d0f27308d66191a6ccc4e9a09a2b3fac2013ba
Nov 01 09:38:00.842411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.842376    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq" event={"ID":"54a81e99-fb8b-4771-a602-1ad94c4420c7","Type":"ContainerStarted","Data":"f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7"}
Nov 01 09:38:00.842907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.842438    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq" event={"ID":"54a81e99-fb8b-4771-a602-1ad94c4420c7","Type":"ContainerStarted","Data":"104f6dca1566f0d88222ad28e6d0f27308d66191a6ccc4e9a09a2b3fac2013ba"}
Nov 01 09:38:00.842907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.842488    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:00.989212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:00.989159    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq" podStartSLOduration=1.9891446799999999 podStartE2EDuration="1.98914468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:00.985642778 +0000 UTC m=+3954.180037334" watchObservedRunningTime="2026-11-01 09:38:00.98914468 +0000 UTC m=+3954.183539235"
Nov 01 09:38:01.702674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.702610    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:38:01.766634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766592    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.766634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766639    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.766921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766677    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.766921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766708    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.766921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766825    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.766921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766890    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.767151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766926    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.767151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767037    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:01.767151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767104    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:01.767329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.766976    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.767329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767281    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:01.767329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767306    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket\") pod \"4a2fd9c0-fdbd-4936-9642-da1174059da4\" (UID: \"4a2fd9c0-fdbd-4936-9642-da1174059da4\") "
Nov 01 09:38:01.767496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767448    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:01.767628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767604    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:01.767977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767954    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.768033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.767986    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.768033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.768004    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.768033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.768018    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2fd9c0-fdbd-4936-9642-da1174059da4-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.768156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.768033    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.769282 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.769253    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:38:01.769450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.769430    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:01.769503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.769467    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "kube-api-access-wkpnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:01.769715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.769690    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy" pod "4a2fd9c0-fdbd-4936-9642-da1174059da4" (UID: "4a2fd9c0-fdbd-4936-9642-da1174059da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:01.852888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.852849    4314 generic.go:356] "Generic (PLEG): container finished" podID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerID="dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7" exitCode=0
Nov 01 09:38:01.853301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.852963    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"
Nov 01 09:38:01.853301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.852979    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" event={"ID":"4a2fd9c0-fdbd-4936-9642-da1174059da4","Type":"ContainerDied","Data":"dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7"}
Nov 01 09:38:01.853301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.853034    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz" event={"ID":"4a2fd9c0-fdbd-4936-9642-da1174059da4","Type":"ContainerDied","Data":"9c4d194ad3767089fb1954aeac77033bd1778212a4fe1e690e00af93e6c97ffe"}
Nov 01 09:38:01.853301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.853057    4314 scope.go:122] "RemoveContainer" containerID="dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7"
Nov 01 09:38:01.865348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.865322    4314 scope.go:122] "RemoveContainer" containerID="3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"
Nov 01 09:38:01.869909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.869870    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkpnd\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-kube-api-access-wkpnd\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.870034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.869911    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.870034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.869929    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.870034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.869942    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2fd9c0-fdbd-4936-9642-da1174059da4-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:01.879819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.879753    4314 scope.go:122] "RemoveContainer" containerID="dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7"
Nov 01 09:38:01.880192 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:01.880154    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7\": container with ID starting with dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7 not found: ID does not exist" containerID="dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7"
Nov 01 09:38:01.880319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.880208    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7"} err="failed to get container status \"dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7\": rpc error: code = NotFound desc = could not find container \"dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7\": container with ID starting with dafc810e76013157bab06331d0a9cd0e58f4b5f9ea473cb2a74ef21d268475c7 not found: ID does not exist"
Nov 01 09:38:01.880319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.880238    4314 scope.go:122] "RemoveContainer" containerID="3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"
Nov 01 09:38:01.880651 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:01.880624    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86\": container with ID starting with 3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86 not found: ID does not exist" containerID="3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"
Nov 01 09:38:01.880710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:01.880660    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86"} err="failed to get container status \"3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86\": rpc error: code = NotFound desc = could not find container \"3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86\": container with ID starting with 3dc0d3b7903c863dd6c26a84cf0d1886a01396af2fdc5df63fea4a7380ac6b86 not found: ID does not exist"
Nov 01 09:38:02.019072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:02.019036    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"]
Nov 01 09:38:02.022939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:02.022913    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-49e170-openshift-default-79b9855b9c-n5wmz"]
Nov 01 09:38:02.543630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:02.543597    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:02.682440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:02.682393    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:02.931151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:02.931070    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" path="/var/lib/kubelet/pods/4a2fd9c0-fdbd-4936-9642-da1174059da4/volumes"
Nov 01 09:38:03.454346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:03.454313    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"]
Nov 01 09:38:03.562480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:03.562444    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"]
Nov 01 09:38:03.613662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:03.613599    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:03.796477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:03.796425    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:03.947853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:03.947822    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:04.712147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:04.712106    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt" podUID="d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" containerName="payload-processing" containerID="cri-o://b2974a2069b927104976be87243f6e1210d8275db30f7588a218229554ee6509" gracePeriod=25
Nov 01 09:38:04.879635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:04.879604    4314 generic.go:356] "Generic (PLEG): container finished" podID="d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" containerID="b2974a2069b927104976be87243f6e1210d8275db30f7588a218229554ee6509" exitCode=0
Nov 01 09:38:04.879837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:04.879688    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt" event={"ID":"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c","Type":"ContainerDied","Data":"b2974a2069b927104976be87243f6e1210d8275db30f7588a218229554ee6509"}
Nov 01 09:38:05.005612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.005585    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:38:05.118093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.118052    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz\") pod \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") "
Nov 01 09:38:05.118297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.118218    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume\") pod \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") "
Nov 01 09:38:05.118297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.118246    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume\") pod \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") "
Nov 01 09:38:05.118420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.118320    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials\") pod \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\" (UID: \"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c\") "
Nov 01 09:38:05.118992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.118759    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume" pod "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" (UID: "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:05.118992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.118865    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume" pod "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" (UID: "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:05.119555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.119512    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:05.119743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.119723    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:05.121178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.121043    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials" pod "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" (UID: "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:05.121178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.121146    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz" pod "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" (UID: "d40d2a68-cc0c-4dd4-9983-67f2763a0e6c"). InnerVolumeSpecName "kube-api-access-t92xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:05.221226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.221188    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-kube-api-access-t92xz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:05.221226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.221223    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:05.887504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.887474    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"
Nov 01 09:38:05.887709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.887507    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-blrnt" event={"ID":"d40d2a68-cc0c-4dd4-9983-67f2763a0e6c","Type":"ContainerDied","Data":"3a33f0d0b1bdefee7a6e1145701a9089164d7fee2a0bfc5526b374356d08a36c"}
Nov 01 09:38:05.887709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:05.887588    4314 scope.go:122] "RemoveContainer" containerID="b2974a2069b927104976be87243f6e1210d8275db30f7588a218229554ee6509"
Nov 01 09:38:06.046077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.046045    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"]
Nov 01 09:38:06.091807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.091776    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-blrnt"]
Nov 01 09:38:06.833033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.832996    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"]
Nov 01 09:38:06.836477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.836453    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:06.866371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.866336    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"]
Nov 01 09:38:06.867028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867008    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" containerName="payload-processing"
Nov 01 09:38:06.867028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867028    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" containerName="payload-processing"
Nov 01 09:38:06.867212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867094    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy"
Nov 01 09:38:06.867212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867103    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy"
Nov 01 09:38:06.867212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867120    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy"
Nov 01 09:38:06.867212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867127    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy"
Nov 01 09:38:06.867422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867265    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy"
Nov 01 09:38:06.867422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867278    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2fd9c0-fdbd-4936-9642-da1174059da4" containerName="istio-proxy"
Nov 01 09:38:06.867422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.867297    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" containerName="payload-processing"
Nov 01 09:38:06.872187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.872166    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:06.929017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.928985    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40d2a68-cc0c-4dd4-9983-67f2763a0e6c" path="/var/lib/kubelet/pods/d40d2a68-cc0c-4dd4-9983-67f2763a0e6c/volumes"
Nov 01 09:38:06.940961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.940932    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"]
Nov 01 09:38:06.942475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.942450    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:06.942622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.942494    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:06.942732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.942708    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5rr\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:06.942821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:06.942806    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.044345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.044296    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.044521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.044477    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.044637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.044520    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.044715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.044680    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5rr\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.045421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.045391    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.045604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.045583    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.047185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.047163    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.102023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.101957    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5rr\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr\") pod \"payload-processing-external-model-85d4446d68-gs2xj\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.186392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.186361    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:07.593226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.593186    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"]
Nov 01 09:38:07.593764 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:07.593736    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec66144_0e29_4f55_b243_954e990edb32.slice/crio-1107dfda05b6a7a63928fac4f8ac8576fda3638a260f7d15a738d29a23777cae WatchSource:0}: Error finding container 1107dfda05b6a7a63928fac4f8ac8576fda3638a260f7d15a738d29a23777cae: Status 404 returned error can't find the container with id 1107dfda05b6a7a63928fac4f8ac8576fda3638a260f7d15a738d29a23777cae
Nov 01 09:38:07.904379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.904289    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" event={"ID":"9ec66144-0e29-4f55-b243-954e990edb32","Type":"ContainerStarted","Data":"297189d71656da532d01ba407bdb8c3f57eaf259110695574223c9acbe58fda8"}
Nov 01 09:38:07.904379 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.904345    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" event={"ID":"9ec66144-0e29-4f55-b243-954e990edb32","Type":"ContainerStarted","Data":"1107dfda05b6a7a63928fac4f8ac8576fda3638a260f7d15a738d29a23777cae"}
Nov 01 09:38:07.904635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:07.904430    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:08.016167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.016112    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" podStartSLOduration=2.016098694 podStartE2EDuration="2.016098694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:08.013018455 +0000 UTC m=+3961.207413010" watchObservedRunningTime="2026-11-01 09:38:08.016098694 +0000 UTC m=+3961.210493248"
Nov 01 09:38:08.455568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.455509    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72" podUID="8f197e1f-d112-4c5c-9036-ca459b164b45" containerName="payload-processing" containerID="cri-o://4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92" gracePeriod=25
Nov 01 09:38:08.562917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.562871    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb" podUID="01d70813-2012-423b-bb37-96b0138fa9d1" containerName="payload-pre-processing" containerID="cri-o://830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602" gracePeriod=25
Nov 01 09:38:08.743961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.743927    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:38:08.805663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.805638    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:38:08.828424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.828373    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:08.867571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.867515    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume\") pod \"8f197e1f-d112-4c5c-9036-ca459b164b45\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") "
Nov 01 09:38:08.867750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.867589    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr\" (UniqueName: \"kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr\") pod \"01d70813-2012-423b-bb37-96b0138fa9d1\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") "
Nov 01 09:38:08.867750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.867623    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume\") pod \"01d70813-2012-423b-bb37-96b0138fa9d1\" (UID: \"01d70813-2012-423b-bb37-96b0138fa9d1\") "
Nov 01 09:38:08.867930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.867914    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd\" (UniqueName: \"kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd\") pod \"8f197e1f-d112-4c5c-9036-ca459b164b45\" (UID: \"8f197e1f-d112-4c5c-9036-ca459b164b45\") "
Nov 01 09:38:08.868007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.867939    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume" pod "8f197e1f-d112-4c5c-9036-ca459b164b45" (UID: "8f197e1f-d112-4c5c-9036-ca459b164b45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:08.868095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.868069    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume" pod "01d70813-2012-423b-bb37-96b0138fa9d1" (UID: "01d70813-2012-423b-bb37-96b0138fa9d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:08.868497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.868469    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f197e1f-d112-4c5c-9036-ca459b164b45-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:08.868639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.868502    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d70813-2012-423b-bb37-96b0138fa9d1-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:08.869867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.869840    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr" pod "01d70813-2012-423b-bb37-96b0138fa9d1" (UID: "01d70813-2012-423b-bb37-96b0138fa9d1"). InnerVolumeSpecName "kube-api-access-grbdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:08.870020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.869997    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd" pod "8f197e1f-d112-4c5c-9036-ca459b164b45" (UID: "8f197e1f-d112-4c5c-9036-ca459b164b45"). InnerVolumeSpecName "kube-api-access-qlqhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:08.913722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.913689    4314 generic.go:356] "Generic (PLEG): container finished" podID="8f197e1f-d112-4c5c-9036-ca459b164b45" containerID="4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92" exitCode=0
Nov 01 09:38:08.913914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.913735    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"
Nov 01 09:38:08.913914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.913770    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72" event={"ID":"8f197e1f-d112-4c5c-9036-ca459b164b45","Type":"ContainerDied","Data":"4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92"}
Nov 01 09:38:08.913914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.913826    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72" event={"ID":"8f197e1f-d112-4c5c-9036-ca459b164b45","Type":"ContainerDied","Data":"e7a8df9a8c3ebced48243c7608a5638a36a0bf21ee46375c8ca0530ce3708e80"}
Nov 01 09:38:08.913914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.913856    4314 scope.go:122] "RemoveContainer" containerID="4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92"
Nov 01 09:38:08.915190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.915171    4314 generic.go:356] "Generic (PLEG): container finished" podID="01d70813-2012-423b-bb37-96b0138fa9d1" containerID="830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602" exitCode=0
Nov 01 09:38:08.915299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.915203    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb" event={"ID":"01d70813-2012-423b-bb37-96b0138fa9d1","Type":"ContainerDied","Data":"830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602"}
Nov 01 09:38:08.915299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.915218    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"
Nov 01 09:38:08.915299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.915246    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb" event={"ID":"01d70813-2012-423b-bb37-96b0138fa9d1","Type":"ContainerDied","Data":"5fef2d6c5f58c94027e8986409de5b519821c70ee43297a7d47cfd77d7d65124"}
Nov 01 09:38:08.924964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.924942    4314 scope.go:122] "RemoveContainer" containerID="4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92"
Nov 01 09:38:08.925262 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:08.925231    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92\": container with ID starting with 4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92 not found: ID does not exist" containerID="4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92"
Nov 01 09:38:08.925366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.925271    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92"} err="failed to get container status \"4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92\": rpc error: code = NotFound desc = could not find container \"4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92\": container with ID starting with 4c02945a8cf01e1fc1664f568917a87d46ebd05bbcb2a94fb79c4face7485f92 not found: ID does not exist"
Nov 01 09:38:08.925366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.925294    4314 scope.go:122] "RemoveContainer" containerID="830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602"
Nov 01 09:38:08.935314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.935298    4314 scope.go:122] "RemoveContainer" containerID="830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602"
Nov 01 09:38:08.935587 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:08.935568    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602\": container with ID starting with 830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602 not found: ID does not exist" containerID="830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602"
Nov 01 09:38:08.935639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.935594    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602"} err="failed to get container status \"830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602\": rpc error: code = NotFound desc = could not find container \"830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602\": container with ID starting with 830ffd278f534b32f81127a3b5720856c8e4e6708b04a4f94f861419543ee602 not found: ID does not exist"
Nov 01 09:38:08.970642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.970573    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlqhd\" (UniqueName: \"kubernetes.io/projected/8f197e1f-d112-4c5c-9036-ca459b164b45-kube-api-access-qlqhd\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:08.970642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.970597    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbdr\" (UniqueName: \"kubernetes.io/projected/01d70813-2012-423b-bb37-96b0138fa9d1-kube-api-access-grbdr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:08.971898 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:08.971874    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:09.081878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.081821    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"]
Nov 01 09:38:09.087407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.087374    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b0d0f3-854fcf46dc-skdgb"]
Nov 01 09:38:09.157331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.157278    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:09.182284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.182252    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"]
Nov 01 09:38:09.185712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.185686    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b0d0f3-66dcf5789f-hpl72"]
Nov 01 09:38:09.610427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.610387    4314 scope.go:122] "RemoveContainer" containerID="e205666664521652a2503209edb6ed976d785029f96219553df2b84c0147eed0"
Nov 01 09:38:09.816880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:09.816844    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:10.109304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:10.109266    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:10.928346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:10.928313    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d70813-2012-423b-bb37-96b0138fa9d1" path="/var/lib/kubelet/pods/01d70813-2012-423b-bb37-96b0138fa9d1/volumes"
Nov 01 09:38:10.928782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:10.928738    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f197e1f-d112-4c5c-9036-ca459b164b45" path="/var/lib/kubelet/pods/8f197e1f-d112-4c5c-9036-ca459b164b45/volumes"
Nov 01 09:38:11.325707 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:11.325676    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:38:11.833826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:11.833783    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq" podUID="54a81e99-fb8b-4771-a602-1ad94c4420c7" containerName="payload-processing" containerID="cri-o://f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7" gracePeriod=25
Nov 01 09:38:12.100560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.100479    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:12.213708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.213673    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials\") pod \"54a81e99-fb8b-4771-a602-1ad94c4420c7\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") "
Nov 01 09:38:12.214150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.213904    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume\") pod \"54a81e99-fb8b-4771-a602-1ad94c4420c7\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") "
Nov 01 09:38:12.214150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.214012    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume\") pod \"54a81e99-fb8b-4771-a602-1ad94c4420c7\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") "
Nov 01 09:38:12.214150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.214053    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn\") pod \"54a81e99-fb8b-4771-a602-1ad94c4420c7\" (UID: \"54a81e99-fb8b-4771-a602-1ad94c4420c7\") "
Nov 01 09:38:12.214411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.214281    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume" pod "54a81e99-fb8b-4771-a602-1ad94c4420c7" (UID: "54a81e99-fb8b-4771-a602-1ad94c4420c7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:12.214738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.214693    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:12.214872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.214790    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume" pod "54a81e99-fb8b-4771-a602-1ad94c4420c7" (UID: "54a81e99-fb8b-4771-a602-1ad94c4420c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:12.216553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.216504    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials" pod "54a81e99-fb8b-4771-a602-1ad94c4420c7" (UID: "54a81e99-fb8b-4771-a602-1ad94c4420c7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:12.216918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.216893    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn" pod "54a81e99-fb8b-4771-a602-1ad94c4420c7" (UID: "54a81e99-fb8b-4771-a602-1ad94c4420c7"). InnerVolumeSpecName "kube-api-access-9sbcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:12.316147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.316117    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:12.316147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.316142    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a81e99-fb8b-4771-a602-1ad94c4420c7-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:12.316147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.316155    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbcn\" (UniqueName: \"kubernetes.io/projected/54a81e99-fb8b-4771-a602-1ad94c4420c7-kube-api-access-9sbcn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:12.757694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.757658    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"]
Nov 01 09:38:12.760493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.760454    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" podUID="9ec66144-0e29-4f55-b243-954e990edb32" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:38:12.787204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787172    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"]
Nov 01 09:38:12.787742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787728    4314 cpu_manager.go:414] "containerMap: removing container" podUID="54a81e99-fb8b-4771-a602-1ad94c4420c7" containerName="payload-processing"
Nov 01 09:38:12.787810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787743    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a81e99-fb8b-4771-a602-1ad94c4420c7" containerName="payload-processing"
Nov 01 09:38:12.787810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787759    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8f197e1f-d112-4c5c-9036-ca459b164b45" containerName="payload-processing"
Nov 01 09:38:12.787810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787765    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f197e1f-d112-4c5c-9036-ca459b164b45" containerName="payload-processing"
Nov 01 09:38:12.787810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787799    4314 cpu_manager.go:414] "containerMap: removing container" podUID="01d70813-2012-423b-bb37-96b0138fa9d1" containerName="payload-pre-processing"
Nov 01 09:38:12.787810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787805    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d70813-2012-423b-bb37-96b0138fa9d1" containerName="payload-pre-processing"
Nov 01 09:38:12.787975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787901    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d70813-2012-423b-bb37-96b0138fa9d1" containerName="payload-pre-processing"
Nov 01 09:38:12.787975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787911    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a81e99-fb8b-4771-a602-1ad94c4420c7" containerName="payload-processing"
Nov 01 09:38:12.787975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.787919    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f197e1f-d112-4c5c-9036-ca459b164b45" containerName="payload-processing"
Nov 01 09:38:12.791371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.791352    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:12.861852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.861816    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"]
Nov 01 09:38:12.923220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.923181    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:12.923401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.923345    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7d2d\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:12.923401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.923385    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:12.923569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.923549    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:12.941548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.941501    4314 generic.go:356] "Generic (PLEG): container finished" podID="54a81e99-fb8b-4771-a602-1ad94c4420c7" containerID="f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7" exitCode=0
Nov 01 09:38:12.941667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.941562    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq" event={"ID":"54a81e99-fb8b-4771-a602-1ad94c4420c7","Type":"ContainerDied","Data":"f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7"}
Nov 01 09:38:12.941667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.941580    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"
Nov 01 09:38:12.941667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.941614    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq" event={"ID":"54a81e99-fb8b-4771-a602-1ad94c4420c7","Type":"ContainerDied","Data":"104f6dca1566f0d88222ad28e6d0f27308d66191a6ccc4e9a09a2b3fac2013ba"}
Nov 01 09:38:12.941667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.941639    4314 scope.go:122] "RemoveContainer" containerID="f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7"
Nov 01 09:38:12.950774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.950755    4314 scope.go:122] "RemoveContainer" containerID="f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7"
Nov 01 09:38:12.951054 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:12.951033    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7\": container with ID starting with f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7 not found: ID does not exist" containerID="f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7"
Nov 01 09:38:12.951107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:12.951063    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7"} err="failed to get container status \"f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7\": rpc error: code = NotFound desc = could not find container \"f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7\": container with ID starting with f1f442e4bb921f7e4461248c209c526cc3ba4acd90a4bb5b88fa53605b6058a7 not found: ID does not exist"
Nov 01 09:38:13.025144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.025051    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.025144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.025137    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.025404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.025298    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7d2d\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.025404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.025351    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.025869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.025842    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.026134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.026112    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.027775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.027756    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.080925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.080841    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"]
Nov 01 09:38:13.084671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.084645    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-f4rbq"]
Nov 01 09:38:13.084837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.084818    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7d2d\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d\") pod \"payload-processing-external-model-7fc854dd88-bl6tf\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") " pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.105544 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.105502    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:13.328168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.328088    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:13.476585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.476558    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"]
Nov 01 09:38:13.477831 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:13.477800    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663923ab_30c3_460b_a78a_fcc3fa220650.slice/crio-558a300c968262c656c755c594d50c953a0e51b5b5b9e160eeb471e48eddd197 WatchSource:0}: Error finding container 558a300c968262c656c755c594d50c953a0e51b5b5b9e160eeb471e48eddd197: Status 404 returned error can't find the container with id 558a300c968262c656c755c594d50c953a0e51b5b5b9e160eeb471e48eddd197
Nov 01 09:38:13.951959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.951920    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf" event={"ID":"663923ab-30c3-460b-a78a-fcc3fa220650","Type":"ContainerStarted","Data":"53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a"}
Nov 01 09:38:13.952178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.951982    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf" event={"ID":"663923ab-30c3-460b-a78a-fcc3fa220650","Type":"ContainerStarted","Data":"558a300c968262c656c755c594d50c953a0e51b5b5b9e160eeb471e48eddd197"}
Nov 01 09:38:13.952178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:13.952010    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:14.045920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:14.045892    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"]
Nov 01 09:38:14.049053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:14.049029    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e652ddf7-m9vzg"]
Nov 01 09:38:14.062864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:14.062820    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf" podStartSLOduration=2.062806202 podStartE2EDuration="2.062806202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:14.060596821 +0000 UTC m=+3967.254991387" watchObservedRunningTime="2026-11-01 09:38:14.062806202 +0000 UTC m=+3967.257200758"
Nov 01 09:38:14.929296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:14.929264    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a81e99-fb8b-4771-a602-1ad94c4420c7" path="/var/lib/kubelet/pods/54a81e99-fb8b-4771-a602-1ad94c4420c7/volumes"
Nov 01 09:38:14.929769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:14.929753    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2" path="/var/lib/kubelet/pods/5af0e5af-0c95-4a87-a7ae-90ae8f0d3aa2/volumes"
Nov 01 09:38:15.067375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:15.067342    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:15.326886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:15.326847    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:15.529488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:15.529443    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:17.509332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.509290    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:17.642179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.642146    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"]
Nov 01 09:38:17.668483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.668446    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:17.679897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.679863    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"]
Nov 01 09:38:17.686065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.686038    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.746125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.746093    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"]
Nov 01 09:38:17.758008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.757959    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" podUID="9ec66144-0e29-4f55-b243-954e990edb32" containerName="payload-processing" containerID="cri-o://297189d71656da532d01ba407bdb8c3f57eaf259110695574223c9acbe58fda8" gracePeriod=25
Nov 01 09:38:17.789818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.789314    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.789818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.789521    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.789818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.789573    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.789818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.789630    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgp2\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.822311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.822277    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:17.901578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.898514    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrgp2\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.901578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.898659    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.901578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.898861    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.901578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.898894    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.901578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.899898    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.901578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.900657    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.931450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.931351    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.957889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.957840    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrgp2\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2\") pod \"payload-processing-external-model-85d4446d68-28gk5\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:17.984920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.984876    4314 generic.go:356] "Generic (PLEG): container finished" podID="9ec66144-0e29-4f55-b243-954e990edb32" containerID="297189d71656da532d01ba407bdb8c3f57eaf259110695574223c9acbe58fda8" exitCode=0
Nov 01 09:38:17.985108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:17.984957    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" event={"ID":"9ec66144-0e29-4f55-b243-954e990edb32","Type":"ContainerDied","Data":"297189d71656da532d01ba407bdb8c3f57eaf259110695574223c9acbe58fda8"}
Nov 01 09:38:18.017551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.012369    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:18.089250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.089227    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:18.210215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.210177    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume\") pod \"9ec66144-0e29-4f55-b243-954e990edb32\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") "
Nov 01 09:38:18.210428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.210322    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume\") pod \"9ec66144-0e29-4f55-b243-954e990edb32\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") "
Nov 01 09:38:18.210428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.210392    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr\") pod \"9ec66144-0e29-4f55-b243-954e990edb32\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") "
Nov 01 09:38:18.210591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.210435    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials\") pod \"9ec66144-0e29-4f55-b243-954e990edb32\" (UID: \"9ec66144-0e29-4f55-b243-954e990edb32\") "
Nov 01 09:38:18.212139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.212106    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume" pod "9ec66144-0e29-4f55-b243-954e990edb32" (UID: "9ec66144-0e29-4f55-b243-954e990edb32"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:18.212350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.212326    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume" pod "9ec66144-0e29-4f55-b243-954e990edb32" (UID: "9ec66144-0e29-4f55-b243-954e990edb32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:18.214034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.214007    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials" pod "9ec66144-0e29-4f55-b243-954e990edb32" (UID: "9ec66144-0e29-4f55-b243-954e990edb32"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:18.215096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.214358    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr" pod "9ec66144-0e29-4f55-b243-954e990edb32" (UID: "9ec66144-0e29-4f55-b243-954e990edb32"). InnerVolumeSpecName "kube-api-access-6j5rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:18.311946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.311910    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:18.311946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.311945    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5rr\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-kube-api-access-6j5rr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:18.311946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.311961    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ec66144-0e29-4f55-b243-954e990edb32-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:18.312312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.311975    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ec66144-0e29-4f55-b243-954e990edb32-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:18.444802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.444704    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"]
Nov 01 09:38:18.452114 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:18.452070    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc79e3a2_0907_4c09_bc06_cbcf61b91506.slice/crio-9ebc69d2c71ba56a6257c6d46c8e91ede6634704a1a10c20827cbc7842d0dce5 WatchSource:0}: Error finding container 9ebc69d2c71ba56a6257c6d46c8e91ede6634704a1a10c20827cbc7842d0dce5: Status 404 returned error can't find the container with id 9ebc69d2c71ba56a6257c6d46c8e91ede6634704a1a10c20827cbc7842d0dce5
Nov 01 09:38:18.513699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:18.513672    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:19.006035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.005984    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj" event={"ID":"9ec66144-0e29-4f55-b243-954e990edb32","Type":"ContainerDied","Data":"1107dfda05b6a7a63928fac4f8ac8576fda3638a260f7d15a738d29a23777cae"}
Nov 01 09:38:19.006293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.006062    4314 scope.go:122] "RemoveContainer" containerID="297189d71656da532d01ba407bdb8c3f57eaf259110695574223c9acbe58fda8"
Nov 01 09:38:19.006293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.006218    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"
Nov 01 09:38:19.010078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.010031    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5" event={"ID":"bc79e3a2-0907-4c09-bc06-cbcf61b91506","Type":"ContainerStarted","Data":"42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10"}
Nov 01 09:38:19.010203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.010105    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5" event={"ID":"bc79e3a2-0907-4c09-bc06-cbcf61b91506","Type":"ContainerStarted","Data":"9ebc69d2c71ba56a6257c6d46c8e91ede6634704a1a10c20827cbc7842d0dce5"}
Nov 01 09:38:19.012864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.012704    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:19.021576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.019566    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"]
Nov 01 09:38:19.021576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.019773    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy" containerID="cri-o://de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05" gracePeriod=30
Nov 01 09:38:19.063476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.063401    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:19.181010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.173218    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"]
Nov 01 09:38:19.181010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.178237    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-gs2xj"]
Nov 01 09:38:19.393620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:19.393497    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5" podStartSLOduration=2.393483956 podStartE2EDuration="2.393483956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:19.39035913 +0000 UTC m=+3972.584753698" watchObservedRunningTime="2026-11-01 09:38:19.393483956 +0000 UTC m=+3972.587878511"
Nov 01 09:38:20.164289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.164254    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"]
Nov 01 09:38:20.199036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.199005    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"]
Nov 01 09:38:20.199624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.199608    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9ec66144-0e29-4f55-b243-954e990edb32" containerName="payload-processing"
Nov 01 09:38:20.199694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.199625    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec66144-0e29-4f55-b243-954e990edb32" containerName="payload-processing"
Nov 01 09:38:20.199756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.199720    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec66144-0e29-4f55-b243-954e990edb32" containerName="payload-processing"
Nov 01 09:38:20.204507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.204490    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.252062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.252029    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.252227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.252091    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.252324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.252227    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj5p\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.252324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.252297    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.257824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.257801    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"]
Nov 01 09:38:20.340036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.340004    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-857c6ebff8fa\""
Nov 01 09:38:20.353227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.353198    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj5p\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.353356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.353252    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.353432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.353373    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.353432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.353424    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.354023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.354000    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.354128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.354066    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.356064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.356039    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.409939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.409914    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj5p\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p\") pod \"payload-processing-external-model-f8b54bcd7-l48d6\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") " pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.518255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.518225    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:20.658587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.658525    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:20.811157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.811070    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:20.893830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.893799    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"]
Nov 01 09:38:20.895371 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:20.895340    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c8e3c1_cfec_4551_a9af_aff97ad8387e.slice/crio-c7fb6730e2155407a9c06cb19e931b8886c6fc3a9655c3c08baf4279bf5e7bc0 WatchSource:0}: Error finding container c7fb6730e2155407a9c06cb19e931b8886c6fc3a9655c3c08baf4279bf5e7bc0: Status 404 returned error can't find the container with id c7fb6730e2155407a9c06cb19e931b8886c6fc3a9655c3c08baf4279bf5e7bc0
Nov 01 09:38:20.929726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:20.929698    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec66144-0e29-4f55-b243-954e990edb32" path="/var/lib/kubelet/pods/9ec66144-0e29-4f55-b243-954e990edb32/volumes"
Nov 01 09:38:21.026843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.026795    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6" event={"ID":"b4c8e3c1-cfec-4551-a9af-aff97ad8387e","Type":"ContainerStarted","Data":"17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d"}
Nov 01 09:38:21.026843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.026849    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6" event={"ID":"b4c8e3c1-cfec-4551-a9af-aff97ad8387e","Type":"ContainerStarted","Data":"c7fb6730e2155407a9c06cb19e931b8886c6fc3a9655c3c08baf4279bf5e7bc0"}
Nov 01 09:38:21.160910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.160797    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6" podStartSLOduration=1.160782445 podStartE2EDuration="1.160782445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:21.158599791 +0000 UTC m=+3974.352994347" watchObservedRunningTime="2026-11-01 09:38:21.160782445 +0000 UTC m=+3974.355177001"
Nov 01 09:38:21.271872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.271840    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"]
Nov 01 09:38:21.276102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.276084    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.328570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.326984    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cddc5b-openshift-default-dockercfg-rr6v2\""
Nov 01 09:38:21.368197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.365033    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"]
Nov 01 09:38:21.385514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.385476    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.385819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.385795    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.385965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.385946    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.386133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.386117    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.386326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.386294    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.386417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.386398    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjbz\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.386481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.386432    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.386549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.386504    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.387014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.386703    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.387233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.387213    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:21.488254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488220    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488312    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488339    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488422    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488452    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488570    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488577    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjbz\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488633    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488717    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488791    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488802    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.488952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.488885    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.489100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.489080    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.489139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.489122    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.490903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.490882    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.491133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.491112    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.545798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.545772    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.545953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.545913    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjbz\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz\") pod \"e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") " pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.589913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.589886    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:21.660089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:21.660029    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:22.013595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.013569    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"]
Nov 01 09:38:22.014429 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:22.014410    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0461a82_08d9_4f66_aa90_c5a5ab48bcf7.slice/crio-b7b086019cb478866c80eb997713612d33d78f8ed569bc2b103bec0639a62b7e WatchSource:0}: Error finding container b7b086019cb478866c80eb997713612d33d78f8ed569bc2b103bec0639a62b7e: Status 404 returned error can't find the container with id b7b086019cb478866c80eb997713612d33d78f8ed569bc2b103bec0639a62b7e
Nov 01 09:38:22.017068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.017031    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:22.017159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.017099    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:22.017159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.017129    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:22.037916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.037883    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" event={"ID":"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7","Type":"ContainerStarted","Data":"b7b086019cb478866c80eb997713612d33d78f8ed569bc2b103bec0639a62b7e"}
Nov 01 09:38:22.038341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.038319    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:22.634555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.634505    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:22.642808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.642777    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf" podUID="663923ab-30c3-460b-a78a-fcc3fa220650" containerName="payload-processing" containerID="cri-o://53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a" gracePeriod=25
Nov 01 09:38:22.901784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:22.901727    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:23.006618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.006593    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d\") pod \"663923ab-30c3-460b-a78a-fcc3fa220650\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") "
Nov 01 09:38:23.006771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.006632    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume\") pod \"663923ab-30c3-460b-a78a-fcc3fa220650\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") "
Nov 01 09:38:23.006771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.006702    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials\") pod \"663923ab-30c3-460b-a78a-fcc3fa220650\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") "
Nov 01 09:38:23.006771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.006727    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume\") pod \"663923ab-30c3-460b-a78a-fcc3fa220650\" (UID: \"663923ab-30c3-460b-a78a-fcc3fa220650\") "
Nov 01 09:38:23.007081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.007050    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume" pod "663923ab-30c3-460b-a78a-fcc3fa220650" (UID: "663923ab-30c3-460b-a78a-fcc3fa220650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:23.007203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.007142    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume" pod "663923ab-30c3-460b-a78a-fcc3fa220650" (UID: "663923ab-30c3-460b-a78a-fcc3fa220650"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:23.008827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.008807    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials" pod "663923ab-30c3-460b-a78a-fcc3fa220650" (UID: "663923ab-30c3-460b-a78a-fcc3fa220650"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:23.008896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.008821    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d" pod "663923ab-30c3-460b-a78a-fcc3fa220650" (UID: "663923ab-30c3-460b-a78a-fcc3fa220650"). InnerVolumeSpecName "kube-api-access-k7d2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:23.046738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.046701    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" event={"ID":"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7","Type":"ContainerStarted","Data":"15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"}
Nov 01 09:38:23.048190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.048163    4314 generic.go:356] "Generic (PLEG): container finished" podID="663923ab-30c3-460b-a78a-fcc3fa220650" containerID="53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a" exitCode=0
Nov 01 09:38:23.048329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.048204    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"
Nov 01 09:38:23.048329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.048242    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf" event={"ID":"663923ab-30c3-460b-a78a-fcc3fa220650","Type":"ContainerDied","Data":"53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a"}
Nov 01 09:38:23.048329 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.048296    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf" event={"ID":"663923ab-30c3-460b-a78a-fcc3fa220650","Type":"ContainerDied","Data":"558a300c968262c656c755c594d50c953a0e51b5b5b9e160eeb471e48eddd197"}
Nov 01 09:38:23.048469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.048332    4314 scope.go:122] "RemoveContainer" containerID="53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a"
Nov 01 09:38:23.059036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.059018    4314 scope.go:122] "RemoveContainer" containerID="53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a"
Nov 01 09:38:23.059299 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:23.059278    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a\": container with ID starting with 53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a not found: ID does not exist" containerID="53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a"
Nov 01 09:38:23.059348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.059307    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a"} err="failed to get container status \"53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a\": rpc error: code = NotFound desc = could not find container \"53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a\": container with ID starting with 53c909e61725a36cd1ba8fa81b2beb5e9f864e6469b6ba2f2b4ed85eff84707a not found: ID does not exist"
Nov 01 09:38:23.108829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.108722    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:23.108829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.108757    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:23.108829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.108773    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7d2d\" (UniqueName: \"kubernetes.io/projected/663923ab-30c3-460b-a78a-fcc3fa220650-kube-api-access-k7d2d\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:23.108829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.108789    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663923ab-30c3-460b-a78a-fcc3fa220650-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:23.164944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.164812    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podStartSLOduration=2.16478768 podStartE2EDuration="2.16478768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:23.158950495 +0000 UTC m=+3976.353345051" watchObservedRunningTime="2026-11-01 09:38:23.16478768 +0000 UTC m=+3976.359182236"
Nov 01 09:38:23.264215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.264175    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"]
Nov 01 09:38:23.280515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.276398    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc854dd88-bl6tf"]
Nov 01 09:38:23.590555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.590503    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:23.596139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.596115    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:23.984094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:23.984064    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:24.057096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.057069    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:24.057976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.057951    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:38:24.295344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.295302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:24.352958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.352933    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:38:24.433319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433287    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433330    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433358    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433384    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433425    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433474    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433551    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433614    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.433788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433646    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:24.433788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433704    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo\") pod \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\" (UID: \"4eb60f2c-7c8d-438b-bde4-45f9034fce60\") "
Nov 01 09:38:24.434014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433788    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:24.434014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433860    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:24.434014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433891    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:24.434014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.433999    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:24.434359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.434338    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.434359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.434359    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.434516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.434368    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.434516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.434376    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.434516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.434389    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.435584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.435559    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:24.435792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.435778    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "kube-api-access-p5xmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:24.435992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.435973    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:38:24.436038 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.436009    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo" pod "4eb60f2c-7c8d-438b-bde4-45f9034fce60" (UID: "4eb60f2c-7c8d-438b-bde4-45f9034fce60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:38:24.535756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.535682    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.535756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.535711    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5xmb\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-kube-api-access-p5xmb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.535756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.535722    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.535756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.535731    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb60f2c-7c8d-438b-bde4-45f9034fce60-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:24.931488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:24.931410    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663923ab-30c3-460b-a78a-fcc3fa220650" path="/var/lib/kubelet/pods/663923ab-30c3-460b-a78a-fcc3fa220650/volumes"
Nov 01 09:38:25.034826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.034787    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"]
Nov 01 09:38:25.037815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.037791    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:25.059743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.059712    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"]
Nov 01 09:38:25.066262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.066235    4314 generic.go:356] "Generic (PLEG): container finished" podID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerID="de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05" exitCode=0
Nov 01 09:38:25.066415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.066291    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"
Nov 01 09:38:25.066415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.066316    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" event={"ID":"4eb60f2c-7c8d-438b-bde4-45f9034fce60","Type":"ContainerDied","Data":"de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05"}
Nov 01 09:38:25.066415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.066368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4" event={"ID":"4eb60f2c-7c8d-438b-bde4-45f9034fce60","Type":"ContainerDied","Data":"6b32f60326b94f4401f0a91f6e4dab26f313b80b491511ec44a91e4a845d5e6b"}
Nov 01 09:38:25.066415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.066394    4314 scope.go:122] "RemoveContainer" containerID="de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05"
Nov 01 09:38:25.078520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.078498    4314 scope.go:122] "RemoveContainer" containerID="31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"
Nov 01 09:38:25.081811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.081785    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3187b904-hg49g"]
Nov 01 09:38:25.085029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085007    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"]
Nov 01 09:38:25.085592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085577    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy"
Nov 01 09:38:25.085647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085593    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy"
Nov 01 09:38:25.085647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085607    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy"
Nov 01 09:38:25.085647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085612    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy"
Nov 01 09:38:25.085647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085626    4314 cpu_manager.go:414] "containerMap: removing container" podUID="663923ab-30c3-460b-a78a-fcc3fa220650" containerName="payload-processing"
Nov 01 09:38:25.085647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085631    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663923ab-30c3-460b-a78a-fcc3fa220650" containerName="payload-processing"
Nov 01 09:38:25.085829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085730    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy"
Nov 01 09:38:25.085829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085741    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" containerName="istio-proxy"
Nov 01 09:38:25.085829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.085755    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="663923ab-30c3-460b-a78a-fcc3fa220650" containerName="payload-processing"
Nov 01 09:38:25.089640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.089616    4314 scope.go:122] "RemoveContainer" containerID="de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05"
Nov 01 09:38:25.089949 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:25.089930    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05\": container with ID starting with de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05 not found: ID does not exist" containerID="de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05"
Nov 01 09:38:25.090023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.089959    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05"} err="failed to get container status \"de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05\": rpc error: code = NotFound desc = could not find container \"de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05\": container with ID starting with de55f72aab27afc3f876f8a8f90a86521265503f59cc3347921d40d42e0f7c05 not found: ID does not exist"
Nov 01 09:38:25.090023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.089976    4314 scope.go:122] "RemoveContainer" containerID="31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"
Nov 01 09:38:25.090307 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:25.090280    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36\": container with ID starting with 31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36 not found: ID does not exist" containerID="31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"
Nov 01 09:38:25.090385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.090315    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36"} err="failed to get container status \"31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36\": rpc error: code = NotFound desc = could not find container \"31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36\": container with ID starting with 31fb1c63fb3937fbe0fba75fc19e26cc9b76796af93b9ba185a5f3d3e309ba36 not found: ID does not exist"
Nov 01 09:38:25.090950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.090926    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.146549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.144409    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.146549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.144493    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.146549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.144782    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggq7s\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.146549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.144829    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.149579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.149556    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"]
Nov 01 09:38:25.245651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.245621    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.245856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.245737    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggq7s\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.245856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.245773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.245978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.245940    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.246427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.246401    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.246681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.246655    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.248682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.248660    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.312077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.312043    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggq7s\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s\") pod \"payload-processing-external-model-85d4446d68-dmgvk\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") " pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.406572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.406515    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:25.528068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.527991    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"]
Nov 01 09:38:25.535990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.535957    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1f3121-openshift-default-797c788959-klwt4"]
Nov 01 09:38:25.775958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.775925    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"]
Nov 01 09:38:25.776645 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:25.776617    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb7ed3b_4486_44b8_8842_8c857457a124.slice/crio-f53c5252e8991d24dc1541fd3dc4267c06c61473d6d8bbbc4d6d9a0cb9a336db WatchSource:0}: Error finding container f53c5252e8991d24dc1541fd3dc4267c06c61473d6d8bbbc4d6d9a0cb9a336db: Status 404 returned error can't find the container with id f53c5252e8991d24dc1541fd3dc4267c06c61473d6d8bbbc4d6d9a0cb9a336db
Nov 01 09:38:25.844825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.844797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:25.991110 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:25.991077    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:26.075202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.075121    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" event={"ID":"cbb7ed3b-4486-44b8-8842-8c857457a124","Type":"ContainerStarted","Data":"606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7"}
Nov 01 09:38:26.075202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.075175    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" event={"ID":"cbb7ed3b-4486-44b8-8842-8c857457a124","Type":"ContainerStarted","Data":"f53c5252e8991d24dc1541fd3dc4267c06c61473d6d8bbbc4d6d9a0cb9a336db"}
Nov 01 09:38:26.075735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.075303    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:38:26.148158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.148129    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:26.185039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.184989    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podStartSLOduration=1.18497468 podStartE2EDuration="1.18497468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:26.184160861 +0000 UTC m=+3979.378555428" watchObservedRunningTime="2026-11-01 09:38:26.18497468 +0000 UTC m=+3979.379369234"
Nov 01 09:38:26.290007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.289975    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:26.929223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.929186    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb60f2c-7c8d-438b-bde4-45f9034fce60" path="/var/lib/kubelet/pods/4eb60f2c-7c8d-438b-bde4-45f9034fce60/volumes"
Nov 01 09:38:26.929755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:26.929731    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801eb4f6-5acf-4720-a0fc-3d84c7127d68" path="/var/lib/kubelet/pods/801eb4f6-5acf-4720-a0fc-3d84c7127d68/volumes"
Nov 01 09:38:27.039083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.039033    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5" podUID="bc79e3a2-0907-4c09-bc06-cbcf61b91506" containerName="payload-processing" containerID="cri-o://42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10" gracePeriod=25
Nov 01 09:38:27.290801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.290775    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:27.367133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.367097    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:27.372313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372285    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume\") pod \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") "
Nov 01 09:38:27.372452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372343    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume\") pod \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") "
Nov 01 09:38:27.372501 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372453    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials\") pod \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") "
Nov 01 09:38:27.372501 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372479    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2\") pod \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\" (UID: \"bc79e3a2-0907-4c09-bc06-cbcf61b91506\") "
Nov 01 09:38:27.372735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372711    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume" pod "bc79e3a2-0907-4c09-bc06-cbcf61b91506" (UID: "bc79e3a2-0907-4c09-bc06-cbcf61b91506"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:27.372799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372774    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume" pod "bc79e3a2-0907-4c09-bc06-cbcf61b91506" (UID: "bc79e3a2-0907-4c09-bc06-cbcf61b91506"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:27.372884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.372868    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:27.374604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.374583    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials" pod "bc79e3a2-0907-4c09-bc06-cbcf61b91506" (UID: "bc79e3a2-0907-4c09-bc06-cbcf61b91506"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:27.374708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.374605    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2" pod "bc79e3a2-0907-4c09-bc06-cbcf61b91506" (UID: "bc79e3a2-0907-4c09-bc06-cbcf61b91506"). InnerVolumeSpecName "kube-api-access-xrgp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:27.473940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.473859    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc79e3a2-0907-4c09-bc06-cbcf61b91506-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:27.473940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.473888    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:27.473940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:27.473898    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrgp2\" (UniqueName: \"kubernetes.io/projected/bc79e3a2-0907-4c09-bc06-cbcf61b91506-kube-api-access-xrgp2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:28.090446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.090411    4314 generic.go:356] "Generic (PLEG): container finished" podID="bc79e3a2-0907-4c09-bc06-cbcf61b91506" containerID="42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10" exitCode=0
Nov 01 09:38:28.090632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.090458    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"
Nov 01 09:38:28.090632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.090491    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5" event={"ID":"bc79e3a2-0907-4c09-bc06-cbcf61b91506","Type":"ContainerDied","Data":"42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10"}
Nov 01 09:38:28.090632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.090578    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-28gk5" event={"ID":"bc79e3a2-0907-4c09-bc06-cbcf61b91506","Type":"ContainerDied","Data":"9ebc69d2c71ba56a6257c6d46c8e91ede6634704a1a10c20827cbc7842d0dce5"}
Nov 01 09:38:28.090632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.090606    4314 scope.go:122] "RemoveContainer" containerID="42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10"
Nov 01 09:38:28.101203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.101183    4314 scope.go:122] "RemoveContainer" containerID="42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10"
Nov 01 09:38:28.101475 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:28.101451    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10\": container with ID starting with 42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10 not found: ID does not exist" containerID="42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10"
Nov 01 09:38:28.101581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.101486    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10"} err="failed to get container status \"42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10\": rpc error: code = NotFound desc = could not find container \"42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10\": container with ID starting with 42d8381ac3810f534c23e3127411a3b4c2af0d822de122fded8fe7c8a2603b10 not found: ID does not exist"
Nov 01 09:38:28.264548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.264497    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"]
Nov 01 09:38:28.265181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.265163    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bc79e3a2-0907-4c09-bc06-cbcf61b91506" containerName="payload-processing"
Nov 01 09:38:28.265245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.265183    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc79e3a2-0907-4c09-bc06-cbcf61b91506" containerName="payload-processing"
Nov 01 09:38:28.265305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.265295    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc79e3a2-0907-4c09-bc06-cbcf61b91506" containerName="payload-processing"
Nov 01 09:38:28.269888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.269859    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.295177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.295150    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"]
Nov 01 09:38:28.305406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.305377    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-28gk5"]
Nov 01 09:38:28.334434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.334311    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:28.372451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.372333    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3a53f2-openshift-default-dockercfg-cm5g5\""
Nov 01 09:38:28.376577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.376550    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"]
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395344    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395408    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395472    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395512    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395730    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395781    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395824    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkv8\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395846    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.398950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.395868    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.496712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.496682    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.496926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.496729    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.496926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.496830    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.496926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.496892    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.496949    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkv8\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.496983    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497017    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497071    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497110    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497152    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497259    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497330    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497424    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.497842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.497823    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.499314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.499284    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.499585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.499563    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.554162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.554130    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.554313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.554277    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkv8\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8\") pod \"e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") " pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.582156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.582121    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:28.585162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.585139    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:28.661495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.661454    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"]
Nov 01 09:38:28.666849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.666821    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.723701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.723672    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"]
Nov 01 09:38:28.800823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.800790    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.800985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.800905    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.800985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.800966    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.801067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.801011    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.801067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.801042    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.801137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.801076    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.801137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.801106    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85n59\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.801201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.801182    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.801239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.801206    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902416    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902470    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902489    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902515    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902549    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902589    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85n59\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902674    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.902804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902709    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.903051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.902814    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.903051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.903017    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.903119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.903049    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.903296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.903275    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.903390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.903344    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.903642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.903625    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.905310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.905291    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.905457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.905442    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.929059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.929035    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc79e3a2-0907-4c09-bc06-cbcf61b91506" path="/var/lib/kubelet/pods/bc79e3a2-0907-4c09-bc06-cbcf61b91506/volumes"
Nov 01 09:38:28.959624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.959600    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.959775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.959759    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85n59\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59\") pod \"maas-default-gateway-openshift-default-746ff8f49f-mhmbr\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:28.982775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:28.982745    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:29.006555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.006517    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"]
Nov 01 09:38:29.006651 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:29.006572    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce0b826_d172_4dc7_9303_d1397880cf52.slice/crio-9076374d7b5d95a96973e7b8dff8bd1f323728908e83c8017aca88a6b0fea693 WatchSource:0}: Error finding container 9076374d7b5d95a96973e7b8dff8bd1f323728908e83c8017aca88a6b0fea693: Status 404 returned error can't find the container with id 9076374d7b5d95a96973e7b8dff8bd1f323728908e83c8017aca88a6b0fea693
Nov 01 09:38:29.009221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.009188    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:29.009295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.009257    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:29.009295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.009287    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:29.101801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.101770    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" event={"ID":"dce0b826-d172-4dc7-9303-d1397880cf52","Type":"ContainerStarted","Data":"9076374d7b5d95a96973e7b8dff8bd1f323728908e83c8017aca88a6b0fea693"}
Nov 01 09:38:29.360485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.360460    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"]
Nov 01 09:38:29.361567 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:29.361516    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf052d21_f5fb_4b68_9f61_705e52111983.slice/crio-bad116392f57d80f1d878da80816c16e5a97ce51cd1d26cabc206f517480cbe7 WatchSource:0}: Error finding container bad116392f57d80f1d878da80816c16e5a97ce51cd1d26cabc206f517480cbe7: Status 404 returned error can't find the container with id bad116392f57d80f1d878da80816c16e5a97ce51cd1d26cabc206f517480cbe7
Nov 01 09:38:29.364728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.364696    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:29.364812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.364756    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:29.364812 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:29.364783    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:38:30.035954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.035916    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6" podUID="b4c8e3c1-cfec-4551-a9af-aff97ad8387e" containerName="payload-processing" containerID="cri-o://17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d" gracePeriod=25
Nov 01 09:38:30.087311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.087280    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:30.112439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.112410    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" event={"ID":"dce0b826-d172-4dc7-9303-d1397880cf52","Type":"ContainerStarted","Data":"4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"}
Nov 01 09:38:30.114187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.114161    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" event={"ID":"bf052d21-f5fb-4b68-9f61-705e52111983","Type":"ContainerStarted","Data":"ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c"}
Nov 01 09:38:30.114338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.114212    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" event={"ID":"bf052d21-f5fb-4b68-9f61-705e52111983","Type":"ContainerStarted","Data":"bad116392f57d80f1d878da80816c16e5a97ce51cd1d26cabc206f517480cbe7"}
Nov 01 09:38:30.247787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.247748    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:30.350074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.350013    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podStartSLOduration=2.349997646 podStartE2EDuration="2.349997646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:30.34606782 +0000 UTC m=+3983.540462376" watchObservedRunningTime="2026-11-01 09:38:30.349997646 +0000 UTC m=+3983.544392200"
Nov 01 09:38:30.350835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.350805    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podStartSLOduration=2.350796317 podStartE2EDuration="2.350796317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:30.232886797 +0000 UTC m=+3983.427281364" watchObservedRunningTime="2026-11-01 09:38:30.350796317 +0000 UTC m=+3983.545190875"
Nov 01 09:38:30.454677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.454648    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:30.499466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.499430    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"]
Nov 01 09:38:30.500120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.500098    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b4c8e3c1-cfec-4551-a9af-aff97ad8387e" containerName="payload-processing"
Nov 01 09:38:30.500249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.500121    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c8e3c1-cfec-4551-a9af-aff97ad8387e" containerName="payload-processing"
Nov 01 09:38:30.500314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.500293    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c8e3c1-cfec-4551-a9af-aff97ad8387e" containerName="payload-processing"
Nov 01 09:38:30.503704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.503683    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.546504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.546473    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials\") pod \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") "
Nov 01 09:38:30.546715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.546628    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p\") pod \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") "
Nov 01 09:38:30.546715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.546651    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume\") pod \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") "
Nov 01 09:38:30.546715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.546712    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume\") pod \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\" (UID: \"b4c8e3c1-cfec-4551-a9af-aff97ad8387e\") "
Nov 01 09:38:30.547147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.547108    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume" pod "b4c8e3c1-cfec-4551-a9af-aff97ad8387e" (UID: "b4c8e3c1-cfec-4551-a9af-aff97ad8387e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:30.547262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.547166    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume" pod "b4c8e3c1-cfec-4551-a9af-aff97ad8387e" (UID: "b4c8e3c1-cfec-4551-a9af-aff97ad8387e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:30.548820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.548798    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p" pod "b4c8e3c1-cfec-4551-a9af-aff97ad8387e" (UID: "b4c8e3c1-cfec-4551-a9af-aff97ad8387e"). InnerVolumeSpecName "kube-api-access-kdj5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:30.548905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.548832    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials" pod "b4c8e3c1-cfec-4551-a9af-aff97ad8387e" (UID: "b4c8e3c1-cfec-4551-a9af-aff97ad8387e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:30.556055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.556030    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"]
Nov 01 09:38:30.556331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.556313    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cddc5b\""
Nov 01 09:38:30.556460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.556442    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cddc5b-dockercfg-dbp4d\""
Nov 01 09:38:30.595555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.593247    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:30.688595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.657410    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.688595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.657584    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtzdl\" (UniqueName: \"kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl\") pod \"payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.688595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.657830    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj5p\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-kube-api-access-kdj5p\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:30.688595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.657850    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:30.688595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.657866    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:30.688595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.657880    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c8e3c1-cfec-4551-a9af-aff97ad8387e-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:30.706193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.706159    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"]
Nov 01 09:38:30.720576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.713908    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:30.720576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.714017    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.748745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.748712    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"]
Nov 01 09:38:30.784941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.784894    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.785146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.785020    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtzdl\" (UniqueName: \"kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl\") pod \"payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.785146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.785103    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwnj\" (UniqueName: \"kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj\") pod \"payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.785313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.785173    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.786280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.786246    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.843738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.843652    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtzdl\" (UniqueName: \"kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl\") pod \"payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:30.886561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.886509    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwnj\" (UniqueName: \"kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj\") pod \"payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.886754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.886616    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.887338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.887316    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.943846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.943816    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwnj\" (UniqueName: \"kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj\") pod \"payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:30.988322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.984036    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:30.988322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.986046    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:30.988322 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:30.986098    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:31.038720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.038686    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:31.084460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.084423    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:31.124817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.120839    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:31.135394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.135312    4314 generic.go:356] "Generic (PLEG): container finished" podID="b4c8e3c1-cfec-4551-a9af-aff97ad8387e" containerID="17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d" exitCode=0
Nov 01 09:38:31.135741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.135690    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6" event={"ID":"b4c8e3c1-cfec-4551-a9af-aff97ad8387e","Type":"ContainerDied","Data":"17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d"}
Nov 01 09:38:31.135799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.135739    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6" event={"ID":"b4c8e3c1-cfec-4551-a9af-aff97ad8387e","Type":"ContainerDied","Data":"c7fb6730e2155407a9c06cb19e931b8886c6fc3a9655c3c08baf4279bf5e7bc0"}
Nov 01 09:38:31.135799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.135771    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"
Nov 01 09:38:31.140792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.136359    4314 scope.go:122] "RemoveContainer" containerID="17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d"
Nov 01 09:38:31.140792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.137774    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:31.141839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.141677    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:38:31.154722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.154685    4314 scope.go:122] "RemoveContainer" containerID="17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d"
Nov 01 09:38:31.155082 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:31.155047    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d\": container with ID starting with 17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d not found: ID does not exist" containerID="17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d"
Nov 01 09:38:31.155175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.155085    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d"} err="failed to get container status \"17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d\": rpc error: code = NotFound desc = could not find container \"17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d\": container with ID starting with 17ccd92c5c948468606878224abb6600b633d31fc711b8eac87505e08aa1568d not found: ID does not exist"
Nov 01 09:38:31.299240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.299143    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"]
Nov 01 09:38:31.302850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.302821    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f8b54bcd7-l48d6"]
Nov 01 09:38:31.448781 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:31.448750    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27ae3d8_2666_4deb_87aa_e9b42aeba696.slice/crio-d06cedf5496a88dc73f64f7855892f06dce8a238c9c58318853f81907a540212 WatchSource:0}: Error finding container d06cedf5496a88dc73f64f7855892f06dce8a238c9c58318853f81907a540212: Status 404 returned error can't find the container with id d06cedf5496a88dc73f64f7855892f06dce8a238c9c58318853f81907a540212
Nov 01 09:38:31.449082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.449054    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"]
Nov 01 09:38:31.546263 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:31.546148    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e50de8_40d1_4fe6_8280_d0a6220657aa.slice/crio-ab54b949f7b3b063c7daecda227f26f6b635c6154c3804d49215e61eafefed6f WatchSource:0}: Error finding container ab54b949f7b3b063c7daecda227f26f6b635c6154c3804d49215e61eafefed6f: Status 404 returned error can't find the container with id ab54b949f7b3b063c7daecda227f26f6b635c6154c3804d49215e61eafefed6f
Nov 01 09:38:31.548410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.548391    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"]
Nov 01 09:38:31.984113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.984071    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:31.984315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:31.984132    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:32.144734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.144699    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd" event={"ID":"45e50de8-40d1-4fe6-8280-d0a6220657aa","Type":"ContainerStarted","Data":"73cc355e7327089a64b249312a61d30565785dac94a29f5487f6f055aa53506b"}
Nov 01 09:38:32.144914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.144749    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd" event={"ID":"45e50de8-40d1-4fe6-8280-d0a6220657aa","Type":"ContainerStarted","Data":"ab54b949f7b3b063c7daecda227f26f6b635c6154c3804d49215e61eafefed6f"}
Nov 01 09:38:32.145005 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.144919    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:32.146499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.146473    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b" event={"ID":"a27ae3d8-2666-4deb-87aa-e9b42aeba696","Type":"ContainerStarted","Data":"cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf"}
Nov 01 09:38:32.146650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.146518    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b" event={"ID":"a27ae3d8-2666-4deb-87aa-e9b42aeba696","Type":"ContainerStarted","Data":"d06cedf5496a88dc73f64f7855892f06dce8a238c9c58318853f81907a540212"}
Nov 01 09:38:32.146650 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.146573    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:32.253584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.253439    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd" podStartSLOduration=2.253423524 podStartE2EDuration="2.253423524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:32.252383269 +0000 UTC m=+3985.446777824" watchObservedRunningTime="2026-11-01 09:38:32.253423524 +0000 UTC m=+3985.447818079"
Nov 01 09:38:32.379557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.379387    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b" podStartSLOduration=2.3793682990000002 podStartE2EDuration="2.379368299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:32.375956465 +0000 UTC m=+3985.570351021" watchObservedRunningTime="2026-11-01 09:38:32.379368299 +0000 UTC m=+3985.573762857"
Nov 01 09:38:32.929453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.929417    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c8e3c1-cfec-4551-a9af-aff97ad8387e" path="/var/lib/kubelet/pods/b4c8e3c1-cfec-4551-a9af-aff97ad8387e/volumes"
Nov 01 09:38:32.984009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.983953    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:32.984209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.984023    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:32.984388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:32.984355    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:33.192652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:33.192544    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:33.394401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:33.394368    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:33.983203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:33.983162    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:33.983654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:33.983224    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:34.984556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:34.984026    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:34.984556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:34.984105    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:35.984165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:35.984126    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:35.984363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:35.984184    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:36.214857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.214803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:36.357706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.357606    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:36.507918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.507864    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:36.627225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.627131    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"]
Nov 01 09:38:36.632622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.632579    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.779598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.773354    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4l6\" (UniqueName: \"kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6\") pod \"payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.788883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.787283    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.791074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.789123    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3a53f2-dockercfg-hsg2r\""
Nov 01 09:38:36.791074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.789211    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3a53f2\""
Nov 01 09:38:36.803592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.798877    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"]
Nov 01 09:38:36.803884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.803850    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"]
Nov 01 09:38:36.809006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.808966    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:36.857006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.856970    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"]
Nov 01 09:38:36.889031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.888885    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4l6\" (UniqueName: \"kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6\") pod \"payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.889250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.889086    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.890139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.890110    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.946964 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.946919    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4l6\" (UniqueName: \"kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6\") pod \"payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.949583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.949546    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:36.983959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.983919    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:36.984140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.984003    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:36.990811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.990761    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pnb\" (UniqueName: \"kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb\") pod \"payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:36.991070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:36.991046    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:37.086589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.086504    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:38:37.092776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.092734    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:37.092990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.092874    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pnb\" (UniqueName: \"kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb\") pod \"payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:37.093704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.093670    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:37.150855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.150773    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pnb\" (UniqueName: \"kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb\") pod \"payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:37.331671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.331633    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"]
Nov 01 09:38:37.332657 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:37.332621    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246bde58_8e09_43e3_b259_7097caf4a021.slice/crio-572e94c4e7c4a2b263877afafc392651d4ec8f6df2bc24d648cd4a4401203c77 WatchSource:0}: Error finding container 572e94c4e7c4a2b263877afafc392651d4ec8f6df2bc24d648cd4a4401203c77: Status 404 returned error can't find the container with id 572e94c4e7c4a2b263877afafc392651d4ec8f6df2bc24d648cd4a4401203c77
Nov 01 09:38:37.424242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.424212    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:37.805900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.805869    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"]
Nov 01 09:38:37.806786 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:38:37.806753    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b33d4_51e4_497b_8b15_0dd05946ae8b.slice/crio-35bbe051393a996c27732e3c375f40a9fb32b4458125091de1dc590d5641cd5d WatchSource:0}: Error finding container 35bbe051393a996c27732e3c375f40a9fb32b4458125091de1dc590d5641cd5d: Status 404 returned error can't find the container with id 35bbe051393a996c27732e3c375f40a9fb32b4458125091de1dc590d5641cd5d
Nov 01 09:38:37.983946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.983903    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-mhmbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 01 09:38:37.984182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:37.983969    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 01 09:38:38.216181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.216077    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6" event={"ID":"246bde58-8e09-43e3-b259-7097caf4a021","Type":"ContainerStarted","Data":"06a3b800afbd1f7c85dfbc791050801e06b07080f0a23c0f289c3d57d4ede483"}
Nov 01 09:38:38.216181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.216139    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6" event={"ID":"246bde58-8e09-43e3-b259-7097caf4a021","Type":"ContainerStarted","Data":"572e94c4e7c4a2b263877afafc392651d4ec8f6df2bc24d648cd4a4401203c77"}
Nov 01 09:38:38.216424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.216182    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:38.218270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.218237    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w" event={"ID":"a53b33d4-51e4-497b-8b15-0dd05946ae8b","Type":"ContainerStarted","Data":"0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd"}
Nov 01 09:38:38.218408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.218296    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w" event={"ID":"a53b33d4-51e4-497b-8b15-0dd05946ae8b","Type":"ContainerStarted","Data":"35bbe051393a996c27732e3c375f40a9fb32b4458125091de1dc590d5641cd5d"}
Nov 01 09:38:38.218408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.218334    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:38.327989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.327917    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6" podStartSLOduration=2.327895208 podStartE2EDuration="2.327895208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:38.323895751 +0000 UTC m=+3991.518290318" watchObservedRunningTime="2026-11-01 09:38:38.327895208 +0000 UTC m=+3991.522289766"
Nov 01 09:38:38.440399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:38.435447    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w" podStartSLOduration=2.435424782 podStartE2EDuration="2.435424782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:38.432173207 +0000 UTC m=+3991.626567763" watchObservedRunningTime="2026-11-01 09:38:38.435424782 +0000 UTC m=+3991.629819339"
Nov 01 09:38:39.407955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:39.407925    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:39.408515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:39.408455    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:39.409288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:39.409266    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:38:41.136635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:41.136601    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:41.324901 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:38:41.324870    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:38:43.157432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:43.157404    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:38:43.157844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:43.157467    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:38:45.980954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:45.980911    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:47.084455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:47.084416    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:38:49.230176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:49.230104    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:38:49.230176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:49.230170    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:38:50.876327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:50.876283    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:55.698353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:55.698312    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:56.520228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:56.520196    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:56.705019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:56.704990    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:38:57.084267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:57.084223    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:38:59.887260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:38:59.887227    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:01.619161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:01.619128    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:01.769961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:01.769923    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:01.836237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:01.836197    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"]
Nov 01 09:39:01.884601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:01.884496    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8071b-5mxxq"]
Nov 01 09:39:02.025296 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:02.025265    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:02.025296 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:02.025293    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:02.025558 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:02.025370    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:02.525350623 +0000 UTC m=+4015.719745175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:02.076317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:02.076272    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"]
Nov 01 09:39:02.076623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:02.076573    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy" containerID="cri-o://2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415" gracePeriod=30
Nov 01 09:39:02.113194 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:02.113166    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:02.531778 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:02.531744    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:02.531778 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:02.531771    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:02.531994 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:02.531833    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:03.531817552 +0000 UTC m=+4016.726212098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:02.929527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:02.929450    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99277be2-353a-4256-a7cd-6bc5f35ac62c" path="/var/lib/kubelet/pods/99277be2-353a-4256-a7cd-6bc5f35ac62c/volumes"
Nov 01 09:39:02.962496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:02.962463    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:03.455382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.455347    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"]
Nov 01 09:39:03.461148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.461130    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.510383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.510347    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-12a01a-openshift-default-dockercfg-x55zq\""
Nov 01 09:39:03.546143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546112    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546330 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546153    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546330 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:03.546318    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:03.546459 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:03.546334    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:03.546459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546353    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546459 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:03.546376    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:05.546363898 +0000 UTC m=+4018.740758435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:03.546459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546404    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546448    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546499    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxkj\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546556    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546657    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.546810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.546718    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.587398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.587362    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"]
Nov 01 09:39:03.596868 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.596832    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.647787    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxkj\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.647840    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.647918    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.647981    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.648119    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.648153    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.648265    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.648306    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.648361    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.651662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.651432    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.652445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.651912    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.652445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.652312    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.652635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.652599    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.653129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.653097    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.658584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.657673    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.691944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.689819    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.714220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.712384    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxkj\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.716361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.715113    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token\") pod \"e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") " pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.775881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.775857    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:03.827547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:03.827494    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:04.204916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.204886    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"]
Nov 01 09:39:04.205930 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:39:04.205903    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbea89c_74ff_4d50_b5c7_8185927cda88.slice/crio-c9c7976797c042f7557835f775e3cee7a83ae4282709e3b4e6bdb1bc8d723bde WatchSource:0}: Error finding container c9c7976797c042f7557835f775e3cee7a83ae4282709e3b4e6bdb1bc8d723bde: Status 404 returned error can't find the container with id c9c7976797c042f7557835f775e3cee7a83ae4282709e3b4e6bdb1bc8d723bde
Nov 01 09:39:04.208658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.208622    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:39:04.208743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.208694    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:39:04.208743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.208728    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:39:04.404954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.404910    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" event={"ID":"fbbea89c-74ff-4d50-b5c7-8185927cda88","Type":"ContainerStarted","Data":"81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"}
Nov 01 09:39:04.405161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.404970    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" event={"ID":"fbbea89c-74ff-4d50-b5c7-8185927cda88","Type":"ContainerStarted","Data":"c9c7976797c042f7557835f775e3cee7a83ae4282709e3b4e6bdb1bc8d723bde"}
Nov 01 09:39:04.520011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.519957    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podStartSLOduration=1.519941819 podStartE2EDuration="1.519941819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:04.517105204 +0000 UTC m=+4017.711499760" watchObservedRunningTime="2026-11-01 09:39:04.519941819 +0000 UTC m=+4017.714336373"
Nov 01 09:39:04.776684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:04.776604    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:05.575640 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:05.575607    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:05.575640 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:05.575632    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:05.576063 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:05.575694    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:09.575678197 +0000 UTC m=+4022.770072750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:05.782978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:05.782950    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:06.419237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:06.419200    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:06.420127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:06.420107    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:39:06.969944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:06.969914    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:07.085973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.085935    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:39:07.339317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.339294    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:39:07.397126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397098    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397148    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397239    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397363    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397394    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397456    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397514    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397569    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397625    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert\") pod \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\" (UID: \"59a5bfcf-d256-4430-99b9-8c5c3d599f4c\") "
Nov 01 09:39:07.397739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397625    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:39:07.397901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397736    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:39:07.397977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.397947    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:39:07.398371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.398288    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:39:07.398484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.398429    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.398484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.398432    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:39:07.398484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.398448    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.398484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.398463    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.398484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.398480    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.400656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.400632    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:39:07.400776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.400691    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "kube-api-access-dcc7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:39:07.400776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.400712    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:39:07.400776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.400722    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy" pod "59a5bfcf-d256-4430-99b9-8c5c3d599f4c" (UID: "59a5bfcf-d256-4430-99b9-8c5c3d599f4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:39:07.427367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.427339    4314 generic.go:356] "Generic (PLEG): container finished" podID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerID="2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415" exitCode=0
Nov 01 09:39:07.427509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.427391    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"
Nov 01 09:39:07.427509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.427428    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" event={"ID":"59a5bfcf-d256-4430-99b9-8c5c3d599f4c","Type":"ContainerDied","Data":"2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415"}
Nov 01 09:39:07.427509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.427499    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8" event={"ID":"59a5bfcf-d256-4430-99b9-8c5c3d599f4c","Type":"ContainerDied","Data":"80067e9b03a91d388db8d80a158fb5a39df652114b757e12567b3f7b5d5f1afd"}
Nov 01 09:39:07.427715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.427554    4314 scope.go:122] "RemoveContainer" containerID="2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415"
Nov 01 09:39:07.438232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.438210    4314 scope.go:122] "RemoveContainer" containerID="0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"
Nov 01 09:39:07.447830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.447811    4314 scope.go:122] "RemoveContainer" containerID="2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415"
Nov 01 09:39:07.448099 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:07.448078    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415\": container with ID starting with 2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415 not found: ID does not exist" containerID="2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415"
Nov 01 09:39:07.448160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.448107    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415"} err="failed to get container status \"2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415\": rpc error: code = NotFound desc = could not find container \"2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415\": container with ID starting with 2189ce2c4f116c0e883a0b99ca6acec1d7091dfd593846e4ed3b8e5c3c0a2415 not found: ID does not exist"
Nov 01 09:39:07.448160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.448124    4314 scope.go:122] "RemoveContainer" containerID="0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"
Nov 01 09:39:07.448357 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:07.448342    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383\": container with ID starting with 0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383 not found: ID does not exist" containerID="0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"
Nov 01 09:39:07.448403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.448360    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383"} err="failed to get container status \"0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383\": rpc error: code = NotFound desc = could not find container \"0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383\": container with ID starting with 0c63c8f531403ba5f66f78b9a9d3e86692de40fe2909a8cc08e8d3a1c3deb383 not found: ID does not exist"
Nov 01 09:39:07.499737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.499658    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.499737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.499685    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.499737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.499700    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcc7b\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-kube-api-access-dcc7b\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.499737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.499713    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.499737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.499725    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5bfcf-d256-4430-99b9-8c5c3d599f4c-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:07.557450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.557418    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"]
Nov 01 09:39:07.561851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.561825    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8071b-openshift-default-579f585c6f-b7xh8"]
Nov 01 09:39:07.697805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.697775    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:07.873222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:07.873140    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:08.025950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:08.025916    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:08.486031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:08.486002    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:08.634662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:08.634629    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:08.930097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:08.930014    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" path="/var/lib/kubelet/pods/59a5bfcf-d256-4430-99b9-8c5c3d599f4c/volumes"
Nov 01 09:39:09.625446 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:09.625417    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:09.625446 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:09.625440    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:09.625881 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:09.625492    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:17.625475075 +0000 UTC m=+4030.819869624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:10.837851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:10.837795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:11.003232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.003199    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:11.330876 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:11.330835    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:39:11.415547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.415497    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"]
Nov 01 09:39:11.417901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.417867    4314 cpu_manager.go:414] "containerMap: removing container" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy"
Nov 01 09:39:11.418103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.418089    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy"
Nov 01 09:39:11.418240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.418222    4314 cpu_manager.go:414] "containerMap: removing container" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy"
Nov 01 09:39:11.418343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.418331    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy"
Nov 01 09:39:11.418672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.418646    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy"
Nov 01 09:39:11.418801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.418790    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a5bfcf-d256-4430-99b9-8c5c3d599f4c" containerName="istio-proxy"
Nov 01 09:39:11.428107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.427880    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.467556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.466079    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"]
Nov 01 09:39:11.505989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.505957    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-12a01a-dockercfg-2slmb\""
Nov 01 09:39:11.505989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.505957    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-12a01a\""
Nov 01 09:39:11.569588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.569516    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnfx\" (UniqueName: \"kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx\") pod \"payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.569758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.569708    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.670503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.670423    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.670697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.670612    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnfx\" (UniqueName: \"kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx\") pod \"payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.671192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.671173    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.728411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.728379    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnfx\" (UniqueName: \"kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx\") pod \"payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:11.762746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:11.762720    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:12.135516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:12.135489    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"]
Nov 01 09:39:12.136364 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:39:12.136337    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0a37f6_2b52_4910_8cf7_0d5971516d82.slice/crio-aababb2722d0821ec2fc2db91587ed29344fb0a4ba7b715e956b076e0ddc7710 WatchSource:0}: Error finding container aababb2722d0821ec2fc2db91587ed29344fb0a4ba7b715e956b076e0ddc7710: Status 404 returned error can't find the container with id aababb2722d0821ec2fc2db91587ed29344fb0a4ba7b715e956b076e0ddc7710
Nov 01 09:39:12.398479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:12.398392    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:12.476262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:12.476217    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s" event={"ID":"1e0a37f6-2b52-4910-8cf7-0d5971516d82","Type":"ContainerStarted","Data":"1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69"}
Nov 01 09:39:12.476505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:12.476294    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:12.476505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:12.476319    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s" event={"ID":"1e0a37f6-2b52-4910-8cf7-0d5971516d82","Type":"ContainerStarted","Data":"aababb2722d0821ec2fc2db91587ed29344fb0a4ba7b715e956b076e0ddc7710"}
Nov 01 09:39:12.586170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:12.586116    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s" podStartSLOduration=1.586102315 podStartE2EDuration="1.586102315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:12.584571081 +0000 UTC m=+4025.778965649" watchObservedRunningTime="2026-11-01 09:39:12.586102315 +0000 UTC m=+4025.780496887"
Nov 01 09:39:16.872400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:16.872363    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:17.084929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:17.084887    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:39:17.649885 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:17.649851    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:17.649885 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:17.649880    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:17.650158 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:17.649933    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:33.649918123 +0000 UTC m=+4046.844312656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:18.818658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:18.818624    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:18.965655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:18.965621    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:19.115900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:19.115814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:20.481271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:20.481239    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:20.968297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:20.968203    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:21.818017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:21.817983    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:22.416730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:22.404341    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:22.612201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:22.612166    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:23.000710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:23.000682    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:23.486066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:23.486036    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:39:24.084470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:24.084438    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:26.462853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:26.462814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:26.630900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:26.630869    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:26.780629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:26.780594    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:27.084172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:27.084081    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:39:29.212944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:29.212906    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:29.581847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:29.581816    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:29.763707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:29.763663    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:30.030293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.030258    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"]
Nov 01 09:39:30.034751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.034728    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.083590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.083520    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49f18f\""
Nov 01 09:39:30.088180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.088147    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"]
Nov 01 09:39:30.114491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.114451    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.114729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.114700    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67dw\" (UniqueName: \"kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw\") pod \"payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.130907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.130838    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49f18f-dockercfg-q4xj8\""
Nov 01 09:39:30.220128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.220074    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67dw\" (UniqueName: \"kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw\") pod \"payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.220949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.220926    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.221927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.221902    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.279148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.278993    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67dw\" (UniqueName: \"kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw\") pod \"payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.349012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.348925    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:30.721914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:30.721881    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"]
Nov 01 09:39:30.722888 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:39:30.722862    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504f61b3_5feb_4422_bb2a_05079ffc1030.slice/crio-cb53276808a2d5c62ff76fcbf8683435cf0d33288fa6e84741eef45d69c04559 WatchSource:0}: Error finding container cb53276808a2d5c62ff76fcbf8683435cf0d33288fa6e84741eef45d69c04559: Status 404 returned error can't find the container with id cb53276808a2d5c62ff76fcbf8683435cf0d33288fa6e84741eef45d69c04559
Nov 01 09:39:31.610706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:31.610667    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2" event={"ID":"504f61b3-5feb-4422-bb2a-05079ffc1030","Type":"ContainerStarted","Data":"323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141"}
Nov 01 09:39:31.611141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:31.610722    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2" event={"ID":"504f61b3-5feb-4422-bb2a-05079ffc1030","Type":"ContainerStarted","Data":"cb53276808a2d5c62ff76fcbf8683435cf0d33288fa6e84741eef45d69c04559"}
Nov 01 09:39:31.611141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:31.610919    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:31.719626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:31.719573    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2" podStartSLOduration=2.7195585060000003 podStartE2EDuration="2.719558506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:31.718841466 +0000 UTC m=+4044.913236021" watchObservedRunningTime="2026-11-01 09:39:31.719558506 +0000 UTC m=+4044.913953058"
Nov 01 09:39:33.673197 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:33.673166    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:33.673197 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:33.673191    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:33.673691 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:33.673255    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:40:05.673239961 +0000 UTC m=+4078.867634495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:39:34.221943 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:34.221907    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:37.084689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:37.084651    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:39:39.045745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:39.045710    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:39.208496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:39.208457    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:39.355595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:39.355475    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:41.323653 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:39:41.323615    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:39:42.621472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:42.621444    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:39:43.980006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:43.979975    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:47.084963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:47.084912    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:39:48.829064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:48.829024    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:51.923025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:51.922989    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:39:53.556142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:53.556105    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:53.715545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:53.715503    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:53.862152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:53.862069    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:58.513721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:58.513682    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:39:59.369071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.369039    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"]
Nov 01 09:39:59.372780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.372762    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.414546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.414508    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"]
Nov 01 09:39:59.418686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.418669    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.448991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.448956    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449053    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449089    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449216    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449299    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449378    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449432    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449518    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449576    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449610    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxsm\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449644    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449681    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449725    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.449807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449767    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449789    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4jn\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449843    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449874    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.449938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.449905    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.505119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.505086    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"]
Nov 01 09:39:59.551608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551570    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551637    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551677    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551702    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxsm\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551826    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551869    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551917    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551961    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.551987    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4jn\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552053    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552065    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552091    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552128    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552229    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552258    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552291    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552350    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552401    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552350    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552524    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552612    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.552813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552614    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.553390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552865    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.553390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.552928    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.553390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.553011    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.553390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.553042    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.553390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.553269    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.553749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.553415    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.554667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.554638    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.555292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.555257    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.555413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.555393    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.555461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.555440    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.582377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.582338    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"]
Nov 01 09:39:59.617900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.616452    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.626161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.626091    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxsm\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.635605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.635566    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vwzjj\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:39:59.635792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.635768    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4jn\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-cqw6z\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.687918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.687882    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:39:59.736186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:39:59.736139    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:40:00.108566 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:40:00.108511    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf126f577_c593_4d2f_ba5a_9ad5edd45429.slice/crio-b9920f58105953c5330f17c9193ca6f144aba8f512988a03eef9c9b75c14ccf9 WatchSource:0}: Error finding container b9920f58105953c5330f17c9193ca6f144aba8f512988a03eef9c9b75c14ccf9: Status 404 returned error can't find the container with id b9920f58105953c5330f17c9193ca6f144aba8f512988a03eef9c9b75c14ccf9
Nov 01 09:40:00.108566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.108522    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"]
Nov 01 09:40:00.115400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.115362    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:00.115512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.115440    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:00.115512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.115472    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:00.158031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.158006    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"]
Nov 01 09:40:00.159023 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:40:00.158994    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009c6f51_7dd6_42ad_8edd_8e07a5772267.slice/crio-2ad4bac6baf02a348909907ea26ab67cf8d5926c42d17a6d0da11ad85c6fc562 WatchSource:0}: Error finding container 2ad4bac6baf02a348909907ea26ab67cf8d5926c42d17a6d0da11ad85c6fc562: Status 404 returned error can't find the container with id 2ad4bac6baf02a348909907ea26ab67cf8d5926c42d17a6d0da11ad85c6fc562
Nov 01 09:40:00.162162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.162127    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:00.162293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.162215    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:00.162293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.162262    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:00.839781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.839729    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" event={"ID":"f126f577-c593-4d2f-ba5a-9ad5edd45429","Type":"ContainerStarted","Data":"272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066"}
Nov 01 09:40:00.840404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.839794    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" event={"ID":"f126f577-c593-4d2f-ba5a-9ad5edd45429","Type":"ContainerStarted","Data":"b9920f58105953c5330f17c9193ca6f144aba8f512988a03eef9c9b75c14ccf9"}
Nov 01 09:40:00.842104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.842073    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" event={"ID":"009c6f51-7dd6-42ad-8edd-8e07a5772267","Type":"ContainerStarted","Data":"13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e"}
Nov 01 09:40:00.842255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:00.842116    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" event={"ID":"009c6f51-7dd6-42ad-8edd-8e07a5772267","Type":"ContainerStarted","Data":"2ad4bac6baf02a348909907ea26ab67cf8d5926c42d17a6d0da11ad85c6fc562"}
Nov 01 09:40:01.009738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.005410    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podStartSLOduration=2.005389546 podStartE2EDuration="2.005389546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:40:00.97952396 +0000 UTC m=+4074.173918517" watchObservedRunningTime="2026-11-01 09:40:01.005389546 +0000 UTC m=+4074.199784102"
Nov 01 09:40:01.111517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.111391    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podStartSLOduration=2.111372557 podStartE2EDuration="2.111372557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:40:01.109217397 +0000 UTC m=+4074.303611948" watchObservedRunningTime="2026-11-01 09:40:01.111372557 +0000 UTC m=+4074.305767112"
Nov 01 09:40:01.693556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.692701    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:40:01.698559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.695581    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:01.698559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.695641    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:01.737599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.737497    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:40:01.739244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.739062    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:01.739244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.739109    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:01.923525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:01.923492    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:40:02.689031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:02.688984    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:02.689362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:02.689328    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:02.737056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:02.737021    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:02.737236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:02.737090    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:03.418284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:03.418130    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:03.692418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:03.692316    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:03.692725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:03.692670    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:03.736723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:03.736695    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:03.736919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:03.736900    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:04.704556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:04.704335    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:04.705107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:04.704513    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:04.736700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:04.736664    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:04.736887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:04.736720    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:05.697558 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:40:05.696828    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:40:05.697558 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:40:05.696871    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:40:05.697558 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:40:05.696951    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:41:09.696916784 +0000 UTC m=+4142.891311323 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:40:05.702566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:05.701677    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:05.702566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:05.701754    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:05.737236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:05.737201    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:05.737642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:05.737253    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:06.689271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:06.689233    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:06.689447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:06.689300    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:06.737491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:06.737452    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:06.737891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:06.737522    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:07.689193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:07.689159    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Nov 01 09:40:07.689381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:07.689223    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Nov 01 09:40:07.736682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:07.736646    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Nov 01 09:40:07.736850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:07.736707    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Nov 01 09:40:08.300199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:08.300161    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:09.714811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.714767    4314 scope.go:122] "RemoveContainer" containerID="d7e20145930c782bfba3f1c59456640f00ac7cda10a7b13df70656919699cde6"
Nov 01 09:40:09.729620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.729525    4314 scope.go:122] "RemoveContainer" containerID="f841dd452436185d22db45fab22147fceeadde933253b03789f99e17fd4f73bb"
Nov 01 09:40:09.731032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.730993    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-cqw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:40:09.731148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.731053    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:40:09.774905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.774506    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vwzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:40:09.774905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.774603    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:40:09.792571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.792213    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:40:09.792992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.792925    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:40:09.794052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.794030    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:40:09.804410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.804387    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:40:09.805593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.805570    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:40:09.805872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:09.805856    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:40:11.107926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:11.107884    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:11.325857 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:40:11.325820    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:40:11.923835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:11.923797    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:40:13.084141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:13.084107    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:17.884034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:17.883997    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:21.923024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:21.922985    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:40:22.669730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:22.669698    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:23.072423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.072337    4314 generic.go:356] "Generic (PLEG): container finished" podID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerID="15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3" exitCode=0
Nov 01 09:40:23.072423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.072374    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" event={"ID":"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7","Type":"ContainerDied","Data":"15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"}
Nov 01 09:40:23.072946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.072794    4314 scope.go:122] "RemoveContainer" containerID="15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"
Nov 01 09:40:23.074419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.074385    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:23.074571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.074471    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:23.074571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.074520    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:40:23.338094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:23.337986    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:24.072556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:24.066658    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:40:24.091004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:24.090967    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" event={"ID":"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7","Type":"ContainerStarted","Data":"2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b"}
Nov 01 09:40:24.590478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:24.590440    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:40:24.591807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:24.591774    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 01 09:40:24.591992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:24.591841    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 01 09:40:25.591136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:25.591096    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 01 09:40:25.591630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:25.591154    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 01 09:40:26.590949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:26.590909    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 01 09:40:26.591135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:26.590966    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 01 09:40:27.419285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:27.419248    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:27.584015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:27.583978    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:27.591219 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:27.591190    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 01 09:40:27.591368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:27.591241    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 01 09:40:27.748846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:27.748819    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:28.590743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:28.590710    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 01 09:40:28.591170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:28.590795    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 01 09:40:29.627781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:29.627741    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:40:29.628612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:29.628583    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:40:29.629894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:29.629873    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:40:31.923297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:31.923262    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:40:32.420586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:32.420525    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:37.161352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:37.161317    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:41.327249 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:40:41.327214    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:40:41.923360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:41.923322    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:40:41.923584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:41.923327    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:46.702120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:46.702078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:48.063693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:48.063662    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:48.207057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:48.207019    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:51.425254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:51.425209    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:51.583751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:51.583718    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:51.726960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:51.726871    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:40:51.923226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:51.923192    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:40:56.413413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:40:56.413376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:01.361477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:01.361423    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:01.923033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:01.923000    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:03.922861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:03.922825    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:06.092793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:06.092758    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:09.738694 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:41:09.738655    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:41:09.738694 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:41:09.738689    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:41:09.739174 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:41:09.738753    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:43:11.738735998 +0000 UTC m=+4264.933130551 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:41:10.989488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:10.989445    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:11.326262 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:41:11.326182    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:41:13.923138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:13.923088    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:14.538173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:14.538139    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:15.833411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:15.833376    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:16.181577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:16.181462    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:16.659895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:16.659859    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:16.809606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:16.809526    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:20.451297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:20.451265    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:20.617152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:20.617116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:20.794123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:20.794087    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:21.163451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:21.163373    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:23.682011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:23.681981    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:23.923122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:23.923087    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:25.275412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.275363    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:25.424589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.424445    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:25.573478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.573401    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:25.681839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.681805    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"]
Nov 01 09:41:25.686139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.686119    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.738849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.738823    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"]
Nov 01 09:41:25.764220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.764185    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2x9\" (UniqueName: \"kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9\") pod \"payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.764455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.764433    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.780617 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.780586    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c81a9d\""
Nov 01 09:41:25.780785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.780706    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c81a9d-dockercfg-95p92\""
Nov 01 09:41:25.867039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.865602    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2x9\" (UniqueName: \"kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9\") pod \"payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.867039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.865802    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.867039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.865982    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"]
Nov 01 09:41:25.867039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.866700    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.874836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.874812    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:25.910380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.910351    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"]
Nov 01 09:41:25.926367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.926327    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2x9\" (UniqueName: \"kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9\") pod \"payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:25.967786 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.967258    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:25.967786 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.967475    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnsjb\" (UniqueName: \"kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb\") pod \"payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:25.999902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:25.999868    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:26.069092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.068928    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnsjb\" (UniqueName: \"kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb\") pod \"payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:26.069189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.069138    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:26.070303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.070270    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:26.128425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.128349    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnsjb\" (UniqueName: \"kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb\") pod \"payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:26.202895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.202858    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:26.389915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.389885    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"]
Nov 01 09:41:26.390299 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:41:26.390020    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec4d3bd_19de_4fd6_a51c_b09baf06d2bf.slice/crio-d66d791ac174aa10914d7ba1626929451c3ffe241674823400c6a829709101e7 WatchSource:0}: Error finding container d66d791ac174aa10914d7ba1626929451c3ffe241674823400c6a829709101e7: Status 404 returned error can't find the container with id d66d791ac174aa10914d7ba1626929451c3ffe241674823400c6a829709101e7
Nov 01 09:41:26.400606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.400579    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:26.588521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.588429    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf" event={"ID":"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf","Type":"ContainerStarted","Data":"77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498"}
Nov 01 09:41:26.588521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.588493    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf" event={"ID":"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf","Type":"ContainerStarted","Data":"d66d791ac174aa10914d7ba1626929451c3ffe241674823400c6a829709101e7"}
Nov 01 09:41:26.588906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.588885    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:26.614262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.614217    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"]
Nov 01 09:41:26.616511 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:41:26.616483    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19919798_47c7_4fb6_9c55_7c42aa189e28.slice/crio-743115c17e8b97f93414928a0a0560a0d180a98ca4e67f9923c6dff8e88f7f1c WatchSource:0}: Error finding container 743115c17e8b97f93414928a0a0560a0d180a98ca4e67f9923c6dff8e88f7f1c: Status 404 returned error can't find the container with id 743115c17e8b97f93414928a0a0560a0d180a98ca4e67f9923c6dff8e88f7f1c
Nov 01 09:41:26.698411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:26.698357    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf" podStartSLOduration=1.698343248 podStartE2EDuration="1.698343248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:41:26.695469902 +0000 UTC m=+4159.889864453" watchObservedRunningTime="2026-11-01 09:41:26.698343248 +0000 UTC m=+4159.892737802"
Nov 01 09:41:27.597351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:27.597310    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb" event={"ID":"19919798-47c7-4fb6-9c55-7c42aa189e28","Type":"ContainerStarted","Data":"a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4"}
Nov 01 09:41:27.597819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:27.597368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb" event={"ID":"19919798-47c7-4fb6-9c55-7c42aa189e28","Type":"ContainerStarted","Data":"743115c17e8b97f93414928a0a0560a0d180a98ca4e67f9923c6dff8e88f7f1c"}
Nov 01 09:41:27.597819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:27.597547    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:27.708262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:27.708208    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb" podStartSLOduration=2.708193627 podStartE2EDuration="2.708193627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:41:27.704862066 +0000 UTC m=+4160.899256622" watchObservedRunningTime="2026-11-01 09:41:27.708193627 +0000 UTC m=+4160.902588181"
Nov 01 09:41:31.560791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:31.560759    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:31.893402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:31.893297    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:32.321828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:32.321796    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:33.173745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:33.173708    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:33.527399 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:33.527366    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:33.923103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:33.923025    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:35.663062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:35.663030    4314 generic.go:356] "Generic (PLEG): container finished" podID="dce0b826-d172-4dc7-9303-d1397880cf52" containerID="4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114" exitCode=0
Nov 01 09:41:35.663559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:35.663107    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" event={"ID":"dce0b826-d172-4dc7-9303-d1397880cf52","Type":"ContainerDied","Data":"4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"}
Nov 01 09:41:35.663640 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:35.663622    4314 scope.go:122] "RemoveContainer" containerID="4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"
Nov 01 09:41:35.665341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:35.665303    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:41:35.665460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:35.665394    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:41:35.665460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:35.665445    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:41:36.689591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:36.689523    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" event={"ID":"dce0b826-d172-4dc7-9303-d1397880cf52","Type":"ContainerStarted","Data":"ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956"}
Nov 01 09:41:36.690188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:36.690158    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:37.527837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.527803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:37.585816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.585778    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:41:37.586004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.585980    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Nov 01 09:41:37.586054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.586033    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Nov 01 09:41:37.600289 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.600256    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:41:37.723243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.723210    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:37.879393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:37.879310    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:38.586380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:38.586340    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Nov 01 09:41:38.586587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:38.586398    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Nov 01 09:41:38.607497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:38.607472    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:41:39.585653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:39.585622    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Nov 01 09:41:39.586089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:39.585671    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Nov 01 09:41:40.586066 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:40.586031    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Nov 01 09:41:40.586544 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:40.586241    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Nov 01 09:41:40.645935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:40.645898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:40.830084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:40.830047    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:41.236947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.236903    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"]
Nov 01 09:41:41.242503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.242476    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.300552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.298882    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b3d6bb\""
Nov 01 09:41:41.301409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.301377    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"]
Nov 01 09:41:41.328974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.328953    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b3d6bb-dockercfg-xjvdt\""
Nov 01 09:41:41.336224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.335966    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.336224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.336089    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wdr\" (UniqueName: \"kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr\") pod \"payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.339307 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:41:41.339283    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:41:41.437884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.437844    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.438064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.438015    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wdr\" (UniqueName: \"kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr\") pod \"payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.438628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.438610    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.496306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.496223    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wdr\" (UniqueName: \"kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr\") pod \"payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.585562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.585510    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Nov 01 09:41:41.585758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.585585    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Nov 01 09:41:41.587791 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.587770    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:41.795133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.795056    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:41.968632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:41.968606    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"]
Nov 01 09:41:41.969485 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:41:41.969457    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25b77bc_54d3_49af_9f2c_a6bedbe8bd65.slice/crio-2bf0c1c94347ae6f0c8efc0f41d99d01bdcede5d08c1e269389337b6c6466adc WatchSource:0}: Error finding container 2bf0c1c94347ae6f0c8efc0f41d99d01bdcede5d08c1e269389337b6c6466adc: Status 404 returned error can't find the container with id 2bf0c1c94347ae6f0c8efc0f41d99d01bdcede5d08c1e269389337b6c6466adc
Nov 01 09:41:42.590249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.590219    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:41:42.590676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.590654    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:41:42.591442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.591420    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:41:42.735577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.735528    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4" event={"ID":"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65","Type":"ContainerStarted","Data":"342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475"}
Nov 01 09:41:42.735753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.735611    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:42.735753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.735633    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4" event={"ID":"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65","Type":"ContainerStarted","Data":"2bf0c1c94347ae6f0c8efc0f41d99d01bdcede5d08c1e269389337b6c6466adc"}
Nov 01 09:41:42.926957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:42.926831    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4" podStartSLOduration=1.926809088 podStartE2EDuration="1.926809088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:41:42.923351394 +0000 UTC m=+4176.117745961" watchObservedRunningTime="2026-11-01 09:41:42.926809088 +0000 UTC m=+4176.121203643"
Nov 01 09:41:43.923053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:43.923021    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:46.613248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:46.613214    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:51.384169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:51.384084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:41:53.746675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:53.746640    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:41:53.923019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:53.922989    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:41:56.153712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:41:56.153668    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:00.982589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:00.982516    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:03.923116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:03.923073    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:42:06.101843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:06.101793    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:06.251774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:06.251739    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:06.570638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:06.570589    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:07.100639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.100605    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"]
Nov 01 09:42:07.105327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.105307    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.232645 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.231713    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f41050-openshift-default-dockercfg-96kx6\""
Nov 01 09:42:07.237562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.237516    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"]
Nov 01 09:42:07.246929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.246896    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.246952    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247269 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247244    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247300    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpt8h\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247339    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247421    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247487    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247558    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.247876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.247768    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.273159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.273123    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:07.348822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.348774    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.349043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.348879    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.349211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349185    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.349297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349270    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349466    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349657    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349714    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpt8h\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349764    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349842    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.349903    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.350261    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.350290    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.350924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.350504    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.351247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.351207    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.354748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.354592    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.355958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.355424    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.407025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.406990    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.407816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.407778    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpt8h\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h\") pod \"e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") " pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.461714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.461679    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:42:07.461910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.461749    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:42:07.547362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.547322    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:07.547815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.547792    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:07.985627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.985593    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"]
Nov 01 09:42:07.985982 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:42:07.985963    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cc2eae_cfd1_4440_af18_46721dd4f8ad.slice/crio-ff83ab7aa3c3e9160597e97b132f21a12254d82bcbfe35853cb6645f25520069 WatchSource:0}: Error finding container ff83ab7aa3c3e9160597e97b132f21a12254d82bcbfe35853cb6645f25520069: Status 404 returned error can't find the container with id ff83ab7aa3c3e9160597e97b132f21a12254d82bcbfe35853cb6645f25520069
Nov 01 09:42:07.988905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.988872    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:42:07.988997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.988934    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:42:07.988997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:07.988965    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:42:08.950372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:08.950332    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" event={"ID":"77cc2eae-cfd1-4440-af18-46721dd4f8ad","Type":"ContainerStarted","Data":"04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"}
Nov 01 09:42:08.950865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:08.950400    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" event={"ID":"77cc2eae-cfd1-4440-af18-46721dd4f8ad","Type":"ContainerStarted","Data":"ff83ab7aa3c3e9160597e97b132f21a12254d82bcbfe35853cb6645f25520069"}
Nov 01 09:42:09.091418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.091364    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podStartSLOduration=2.0913496 podStartE2EDuration="2.0913496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:42:09.086708556 +0000 UTC m=+4202.281103136" watchObservedRunningTime="2026-11-01 09:42:09.0913496 +0000 UTC m=+4202.285744155"
Nov 01 09:42:09.549175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.549098    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:09.555107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.555080    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:09.786107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.786078    4314 scope.go:122] "RemoveContainer" containerID="495ae6aee441d9f2524dfb4183ba963eaf21ce3e44220203c1a7fdf0dbc034f7"
Nov 01 09:42:09.796710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.796687    4314 scope.go:122] "RemoveContainer" containerID="5b86b4324f26077690c03e40ea5092851828bfdc380bff22b9bad9663a053e64"
Nov 01 09:42:09.958623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.958593    4314 generic.go:356] "Generic (PLEG): container finished" podID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerID="81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e" exitCode=0
Nov 01 09:42:09.959046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.958669    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" event={"ID":"fbbea89c-74ff-4d50-b5c7-8185927cda88","Type":"ContainerDied","Data":"81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"}
Nov 01 09:42:09.959100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.959049    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:09.959148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.959096    4314 scope.go:122] "RemoveContainer" containerID="81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"
Nov 01 09:42:09.960173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.960152    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:42:09.960735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.960710    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:42:09.960794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.960776    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:42:09.960831 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:09.960814    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:42:10.988268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:10.988230    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" event={"ID":"fbbea89c-74ff-4d50-b5c7-8185927cda88","Type":"ContainerStarted","Data":"43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715"}
Nov 01 09:42:11.325333 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:42:11.325249    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:42:11.378377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:11.378159    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:11.778555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:11.777897    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:42:11.778555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:11.778021    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Nov 01 09:42:11.778555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:11.778064    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Nov 01 09:42:11.899827 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:11.899792    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:12.078108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:12.078023    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:12.235699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:12.235655    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:12.780556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:12.779921    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Nov 01 09:42:12.780556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:12.779986    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Nov 01 09:42:13.782579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:13.778916    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Nov 01 09:42:13.782579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:13.778978    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Nov 01 09:42:13.923733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:13.923695    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:42:14.782027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:14.781990    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Nov 01 09:42:14.782324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:14.782295    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Nov 01 09:42:15.081447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:15.081354    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:15.239701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:15.239667    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:15.782639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:15.781950    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Nov 01 09:42:15.782639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:15.782056    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Nov 01 09:42:16.599671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:16.599634    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:16.781161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:16.781122    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:42:16.781585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:16.781564    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:42:16.782251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:16.782231    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:42:21.380576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:21.380529    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:23.923521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:23.923483    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:42:24.036302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:24.036249    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:26.155545 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:26.155507    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:26.927323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:26.927287    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:42:30.446334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:30.446298    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:31.431826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:31.431797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:36.451023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:36.450987    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:36.626995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:36.626956    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:36.850951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:36.850909    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"]
Nov 01 09:42:36.927826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:36.927783    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:42:37.097294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:37.097257    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:37.246935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:37.246870    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:41.325290 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:42:41.325251    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:42:41.677314 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:41.677213    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:41.852177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:41.852115    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2" podUID="504f61b3-5feb-4422-bb2a-05079ffc1030" containerName="payload-pre-processing" containerID="cri-o://323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141" gracePeriod=25
Nov 01 09:42:42.130445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.130420    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:42:42.212448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.212411    4314 generic.go:356] "Generic (PLEG): container finished" podID="504f61b3-5feb-4422-bb2a-05079ffc1030" containerID="323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141" exitCode=0
Nov 01 09:42:42.212655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.212454    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"
Nov 01 09:42:42.212655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.212495    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2" event={"ID":"504f61b3-5feb-4422-bb2a-05079ffc1030","Type":"ContainerDied","Data":"323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141"}
Nov 01 09:42:42.212655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.212583    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2" event={"ID":"504f61b3-5feb-4422-bb2a-05079ffc1030","Type":"ContainerDied","Data":"cb53276808a2d5c62ff76fcbf8683435cf0d33288fa6e84741eef45d69c04559"}
Nov 01 09:42:42.212655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.212619    4314 scope.go:122] "RemoveContainer" containerID="323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141"
Nov 01 09:42:42.222568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.222550    4314 scope.go:122] "RemoveContainer" containerID="323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141"
Nov 01 09:42:42.222831 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:42:42.222809    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141\": container with ID starting with 323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141 not found: ID does not exist" containerID="323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141"
Nov 01 09:42:42.222874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.222838    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141"} err="failed to get container status \"323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141\": rpc error: code = NotFound desc = could not find container \"323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141\": container with ID starting with 323e04e473d901d0e84a41e965b1f7832d018dfee1d6e85c5eb9eaf189dc3141 not found: ID does not exist"
Nov 01 09:42:42.272594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.269917    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw\" (UniqueName: \"kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw\") pod \"504f61b3-5feb-4422-bb2a-05079ffc1030\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") "
Nov 01 09:42:42.272594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.270097    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume\") pod \"504f61b3-5feb-4422-bb2a-05079ffc1030\" (UID: \"504f61b3-5feb-4422-bb2a-05079ffc1030\") "
Nov 01 09:42:42.272594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.270672    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume" pod "504f61b3-5feb-4422-bb2a-05079ffc1030" (UID: "504f61b3-5feb-4422-bb2a-05079ffc1030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:42:42.272912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.272816    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw" pod "504f61b3-5feb-4422-bb2a-05079ffc1030" (UID: "504f61b3-5feb-4422-bb2a-05079ffc1030"). InnerVolumeSpecName "kube-api-access-t67dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:42:42.309501 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.309469    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:42.372052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.372019    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504f61b3-5feb-4422-bb2a-05079ffc1030-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:42:42.372052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.372048    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67dw\" (UniqueName: \"kubernetes.io/projected/504f61b3-5feb-4422-bb2a-05079ffc1030-kube-api-access-t67dw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:42:42.627083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.627052    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"]
Nov 01 09:42:42.673413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.673379    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49f18f-df46c885f-ffzz2"]
Nov 01 09:42:42.930323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:42.930247    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504f61b3-5feb-4422-bb2a-05079ffc1030" path="/var/lib/kubelet/pods/504f61b3-5feb-4422-bb2a-05079ffc1030/volumes"
Nov 01 09:42:45.057054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:45.057019    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"]
Nov 01 09:42:45.102746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:45.102711    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a82c5-xs8z2"]
Nov 01 09:42:46.806097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:46.806063    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:46.927303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:46.927269    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:42:46.930979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:46.930957    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb" path="/var/lib/kubelet/pods/b6e28ba4-97f9-454a-8f7b-cfb0d09abbeb/volumes"
Nov 01 09:42:51.741816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:51.741781    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:51.888714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:51.888674    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:52.038348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:52.038263    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:56.782183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:56.782147    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:42:56.930615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:42:56.930577    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:01.548331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:01.548282    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:04.264081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:04.264038    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:04.411309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:04.411224    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:06.361257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:06.361197    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:06.927435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:06.927396    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:11.172824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:11.172788    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:11.324698 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:43:11.324665    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:43:11.343782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:11.343749    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:11.492365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:11.492290    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:11.781295 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:43:11.781213    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:43:11.781295 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:43:11.781244    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: secret "agc-cred-47f306d7be56" not found
Nov 01 09:43:11.781500 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:43:11.781308    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:45:13.781292639 +0000 UTC m=+4386.975687189 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : secret "agc-cred-47f306d7be56" not found
Nov 01 09:43:16.280882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:16.280850    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:16.927171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:16.927134    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:21.199170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:21.199090    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:24.390498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:24.390463    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:26.056187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:26.056138    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:26.927273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:26.927231    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:29.307203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:29.307166    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:29.470721 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:29.470689    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:30.904211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:30.904176    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:35.819752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:35.819717    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:36.927348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:36.927313    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:40.577698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:40.577663    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:40.922569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:40.922464    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:41.324676 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:43:41.324641    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:43:45.393974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:45.393937    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:50.131033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:50.130998    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:50.295287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:50.295252    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:50.451343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:50.451247    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:50.922585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:50.922519    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:43:53.549856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:53.549825    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:53.698772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:53.698735    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:55.125919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:55.125880    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:43:59.894186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:43:59.894153    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:00.923030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:00.922974    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:01.139798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.139764    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"]
Nov 01 09:44:01.140446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.140432    4314 cpu_manager.go:414] "containerMap: removing container" podUID="504f61b3-5feb-4422-bb2a-05079ffc1030" containerName="payload-pre-processing"
Nov 01 09:44:01.140495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.140448    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504f61b3-5feb-4422-bb2a-05079ffc1030" containerName="payload-pre-processing"
Nov 01 09:44:01.140603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.140592    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="504f61b3-5feb-4422-bb2a-05079ffc1030" containerName="payload-pre-processing"
Nov 01 09:44:01.144237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.144223    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.197892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.197821    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"]
Nov 01 09:44:01.239231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.239200    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:44:01.239424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.239235    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:44:01.239424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.239283    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:44:01.239424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.239374    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:44:01.255124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.255099    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.255240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.255143    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.255240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.255161    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnpv\" (UniqueName: \"kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.356729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.356695    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.356899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.356753    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.356899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.356778    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnpv\" (UniqueName: \"kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.357437 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.357416    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.359097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.359075    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.414247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.414209    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnpv\" (UniqueName: \"kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv\") pod \"maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.458864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.458799    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:01.828774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:01.828743    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"]
Nov 01 09:44:01.830215 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:44:01.830180    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf899b215_83e5_4a09_842a_9b641615f02d.slice/crio-f92ea282f4054faac20ef2d5e7f1253f193498cee48221485cceccb628facc07 WatchSource:0}: Error finding container f92ea282f4054faac20ef2d5e7f1253f193498cee48221485cceccb628facc07: Status 404 returned error can't find the container with id f92ea282f4054faac20ef2d5e7f1253f193498cee48221485cceccb628facc07
Nov 01 09:44:02.792243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:02.792215    4314 generic.go:356] "Generic (PLEG): container finished" podID="f899b215-83e5-4a09-842a-9b641615f02d" containerID="79cc6882bd41c925056320b3dcae203de6812361e3bad6db3e0da9ca0059a65f" exitCode=0
Nov 01 09:44:02.792613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:02.792299    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks" event={"ID":"f899b215-83e5-4a09-842a-9b641615f02d","Type":"ContainerDied","Data":"79cc6882bd41c925056320b3dcae203de6812361e3bad6db3e0da9ca0059a65f"}
Nov 01 09:44:02.792613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:02.792352    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks" event={"ID":"f899b215-83e5-4a09-842a-9b641615f02d","Type":"ContainerStarted","Data":"f92ea282f4054faac20ef2d5e7f1253f193498cee48221485cceccb628facc07"}
Nov 01 09:44:03.966300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:03.966273    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:04.092256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.092182    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca\") pod \"f899b215-83e5-4a09-842a-9b641615f02d\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") "
Nov 01 09:44:04.092436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.092414    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca\") pod \"f899b215-83e5-4a09-842a-9b641615f02d\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") "
Nov 01 09:44:04.092525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.092469    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv\" (UniqueName: \"kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv\") pod \"f899b215-83e5-4a09-842a-9b641615f02d\" (UID: \"f899b215-83e5-4a09-842a-9b641615f02d\") "
Nov 01 09:44:04.093118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.092761    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca" pod "f899b215-83e5-4a09-842a-9b641615f02d" (UID: "f899b215-83e5-4a09-842a-9b641615f02d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:44:04.093118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.093089    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:04.094737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.094706    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv" pod "f899b215-83e5-4a09-842a-9b641615f02d" (UID: "f899b215-83e5-4a09-842a-9b641615f02d"). InnerVolumeSpecName "kube-api-access-xjnpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:44:04.094955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.094930    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca" pod "f899b215-83e5-4a09-842a-9b641615f02d" (UID: "f899b215-83e5-4a09-842a-9b641615f02d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:44:04.194121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.194089    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnpv\" (UniqueName: \"kubernetes.io/projected/f899b215-83e5-4a09-842a-9b641615f02d-kube-api-access-xjnpv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:04.194121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.194117    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f899b215-83e5-4a09-842a-9b641615f02d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:04.807566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.807525    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"
Nov 01 09:44:04.807756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.807565    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks" event={"ID":"f899b215-83e5-4a09-842a-9b641615f02d","Type":"ContainerDied","Data":"f92ea282f4054faac20ef2d5e7f1253f193498cee48221485cceccb628facc07"}
Nov 01 09:44:04.807756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:04.807625    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f92ea282f4054faac20ef2d5e7f1253f193498cee48221485cceccb628facc07"
Nov 01 09:44:05.186463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:05.186381    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:09.843362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:09.843331    4314 scope.go:122] "RemoveContainer" containerID="270ad5c85032e7defda08e1c24dc24c48c63bf48403237d680922d5202cd498b"
Nov 01 09:44:09.854156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:09.854136    4314 scope.go:122] "RemoveContainer" containerID="54b59ef8bd0df9c8c1a5b01d1429141dd0db924f66a213633d56ed848e4bcb0f"
Nov 01 09:44:10.040998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:10.040965    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:10.922699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:10.922660    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:11.338208 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:44:11.338159    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:44:14.804809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:14.804778    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:18.934752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:18.934672    4314 generic.go:356] "Generic (PLEG): container finished" podID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerID="04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e" exitCode=0
Nov 01 09:44:18.934752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:18.934718    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" event={"ID":"77cc2eae-cfd1-4440-af18-46721dd4f8ad","Type":"ContainerDied","Data":"04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"}
Nov 01 09:44:18.935244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:18.935063    4314 scope.go:122] "RemoveContainer" containerID="04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"
Nov 01 09:44:18.936816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:18.936786    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:44:18.936903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:18.936856    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:44:18.936903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:18.936890    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:44:19.362702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:19.362672    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:19.549345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:19.549312    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:19.944357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:19.944320    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" event={"ID":"77cc2eae-cfd1-4440-af18-46721dd4f8ad","Type":"ContainerStarted","Data":"137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43"}
Nov 01 09:44:20.549016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:20.548980    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:44:20.549927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:20.549901    4314 patch_prober.go:29] interesting pod/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Nov 01 09:44:20.550047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:20.549949    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Nov 01 09:44:20.922578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:20.922462    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:21.549784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:21.549749    4314 patch_prober.go:29] interesting pod/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Nov 01 09:44:21.550191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:21.549803    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Nov 01 09:44:22.549506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:22.549472    4314 patch_prober.go:29] interesting pod/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Nov 01 09:44:22.549717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:22.549555    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Nov 01 09:44:23.549926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:23.549888    4314 patch_prober.go:29] interesting pod/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Nov 01 09:44:23.550389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:23.549954    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Nov 01 09:44:24.303771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:24.303740    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:24.549108 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:24.549075    4314 patch_prober.go:29] interesting pod/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Nov 01 09:44:24.549308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:24.549148    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Nov 01 09:44:24.721901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:24.721823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:24.959678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:24.959645    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:44:25.276771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:25.276711    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:25.553883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:25.553809    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:44:25.554909 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:25.554886    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:44:25.689742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:25.689703    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"]
Nov 01 09:44:25.780771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:25.780742    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"]
Nov 01 09:44:26.024826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:26.024795    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:26.168773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:26.168737    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:29.483262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:29.483232    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:30.690629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:30.690563    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b" podUID="a27ae3d8-2666-4deb-87aa-e9b42aeba696" containerName="payload-processing" containerID="cri-o://cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf" gracePeriod=25
Nov 01 09:44:30.781041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:30.781002    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd" podUID="45e50de8-40d1-4fe6-8280-d0a6220657aa" containerName="payload-pre-processing" containerID="cri-o://73cc355e7327089a64b249312a61d30565785dac94a29f5487f6f055aa53506b" gracePeriod=25
Nov 01 09:44:30.923792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:30.923753    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:31.051554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.051512    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:44:31.075335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.075307    4314 generic.go:356] "Generic (PLEG): container finished" podID="45e50de8-40d1-4fe6-8280-d0a6220657aa" containerID="73cc355e7327089a64b249312a61d30565785dac94a29f5487f6f055aa53506b" exitCode=0
Nov 01 09:44:31.075502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.075387    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd" event={"ID":"45e50de8-40d1-4fe6-8280-d0a6220657aa","Type":"ContainerDied","Data":"73cc355e7327089a64b249312a61d30565785dac94a29f5487f6f055aa53506b"}
Nov 01 09:44:31.076751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.076727    4314 generic.go:356] "Generic (PLEG): container finished" podID="a27ae3d8-2666-4deb-87aa-e9b42aeba696" containerID="cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf" exitCode=0
Nov 01 09:44:31.076872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.076773    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"
Nov 01 09:44:31.076872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.076811    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b" event={"ID":"a27ae3d8-2666-4deb-87aa-e9b42aeba696","Type":"ContainerDied","Data":"cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf"}
Nov 01 09:44:31.076872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.076860    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b" event={"ID":"a27ae3d8-2666-4deb-87aa-e9b42aeba696","Type":"ContainerDied","Data":"d06cedf5496a88dc73f64f7855892f06dce8a238c9c58318853f81907a540212"}
Nov 01 09:44:31.077011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.076883    4314 scope.go:122] "RemoveContainer" containerID="cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf"
Nov 01 09:44:31.088975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.088955    4314 scope.go:122] "RemoveContainer" containerID="cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf"
Nov 01 09:44:31.089239 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:44:31.089221    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf\": container with ID starting with cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf not found: ID does not exist" containerID="cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf"
Nov 01 09:44:31.089298 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.089251    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf"} err="failed to get container status \"cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf\": rpc error: code = NotFound desc = could not find container \"cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf\": container with ID starting with cae9156dd531db19b7333bf54ce36c844b26b4c8002e6cf6b3519dfea5eb9adf not found: ID does not exist"
Nov 01 09:44:31.095773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.095754    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj\" (UniqueName: \"kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj\") pod \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") "
Nov 01 09:44:31.095933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.095913    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume\") pod \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\" (UID: \"a27ae3d8-2666-4deb-87aa-e9b42aeba696\") "
Nov 01 09:44:31.096247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.096225    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume" pod "a27ae3d8-2666-4deb-87aa-e9b42aeba696" (UID: "a27ae3d8-2666-4deb-87aa-e9b42aeba696"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:44:31.096485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.096471    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27ae3d8-2666-4deb-87aa-e9b42aeba696-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:31.097738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.097719    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj" pod "a27ae3d8-2666-4deb-87aa-e9b42aeba696" (UID: "a27ae3d8-2666-4deb-87aa-e9b42aeba696"). InnerVolumeSpecName "kube-api-access-dkwnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:44:31.106044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.106025    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:44:31.152566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.152525    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:31.197891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.197851    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl\" (UniqueName: \"kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl\") pod \"45e50de8-40d1-4fe6-8280-d0a6220657aa\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") "
Nov 01 09:44:31.198094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.197966    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume\") pod \"45e50de8-40d1-4fe6-8280-d0a6220657aa\" (UID: \"45e50de8-40d1-4fe6-8280-d0a6220657aa\") "
Nov 01 09:44:31.198348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.198310    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume" pod "45e50de8-40d1-4fe6-8280-d0a6220657aa" (UID: "45e50de8-40d1-4fe6-8280-d0a6220657aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:44:31.198826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.198799    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkwnj\" (UniqueName: \"kubernetes.io/projected/a27ae3d8-2666-4deb-87aa-e9b42aeba696-kube-api-access-dkwnj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:31.198940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.198832    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e50de8-40d1-4fe6-8280-d0a6220657aa-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:31.200019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.199989    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl" pod "45e50de8-40d1-4fe6-8280-d0a6220657aa" (UID: "45e50de8-40d1-4fe6-8280-d0a6220657aa"). InnerVolumeSpecName "kube-api-access-wtzdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:44:31.300543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.300503    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtzdl\" (UniqueName: \"kubernetes.io/projected/45e50de8-40d1-4fe6-8280-d0a6220657aa-kube-api-access-wtzdl\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:31.499039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.499004    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"]
Nov 01 09:44:31.502951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:31.502923    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cddc5b-6ffc6c5ff7-r946b"]
Nov 01 09:44:32.084688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.084646    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd" event={"ID":"45e50de8-40d1-4fe6-8280-d0a6220657aa","Type":"ContainerDied","Data":"ab54b949f7b3b063c7daecda227f26f6b635c6154c3804d49215e61eafefed6f"}
Nov 01 09:44:32.085149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.084722    4314 scope.go:122] "RemoveContainer" containerID="73cc355e7327089a64b249312a61d30565785dac94a29f5487f6f055aa53506b"
Nov 01 09:44:32.085149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.084661    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"
Nov 01 09:44:32.239588 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.239504    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"]
Nov 01 09:44:32.285454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.285424    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cddc5b-84b5b4764c-z22sd"]
Nov 01 09:44:32.930320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.930287    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e50de8-40d1-4fe6-8280-d0a6220657aa" path="/var/lib/kubelet/pods/45e50de8-40d1-4fe6-8280-d0a6220657aa/volumes"
Nov 01 09:44:32.930758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:32.930743    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27ae3d8-2666-4deb-87aa-e9b42aeba696" path="/var/lib/kubelet/pods/a27ae3d8-2666-4deb-87aa-e9b42aeba696/volumes"
Nov 01 09:44:34.788803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:34.788773    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:37.571769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:37.571737    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:37.711328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:37.711296    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:39.966511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:39.966476    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:40.922738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:40.922698    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:41.323795 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:44:41.323758    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:44:41.456529 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:41.456496    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:41.922884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:41.922845    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:44.772934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:44.772901    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:47.452250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:47.452210    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:47.901647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:47.901610    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"]
Nov 01 09:44:48.002670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:48.002641    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"]
Nov 01 09:44:48.301448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:48.301410    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:48.444362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:48.444325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:49.811930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:49.811891    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:51.923698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:51.923659    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:44:52.902630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:52.902593    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w" podUID="a53b33d4-51e4-497b-8b15-0dd05946ae8b" containerName="payload-processing" containerID="cri-o://0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd" gracePeriod=25
Nov 01 09:44:53.003293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.003245    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6" podUID="246bde58-8e09-43e3-b259-7097caf4a021" containerName="payload-pre-processing" containerID="cri-o://06a3b800afbd1f7c85dfbc791050801e06b07080f0a23c0f289c3d57d4ede483" gracePeriod=25
Nov 01 09:44:53.193053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.192693    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:44:53.254908 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.254872    4314 generic.go:356] "Generic (PLEG): container finished" podID="a53b33d4-51e4-497b-8b15-0dd05946ae8b" containerID="0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd" exitCode=0
Nov 01 09:44:53.255122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.254953    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"
Nov 01 09:44:53.255122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.254953    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w" event={"ID":"a53b33d4-51e4-497b-8b15-0dd05946ae8b","Type":"ContainerDied","Data":"0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd"}
Nov 01 09:44:53.255233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.255128    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w" event={"ID":"a53b33d4-51e4-497b-8b15-0dd05946ae8b","Type":"ContainerDied","Data":"35bbe051393a996c27732e3c375f40a9fb32b4458125091de1dc590d5641cd5d"}
Nov 01 09:44:53.255233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.255179    4314 scope.go:122] "RemoveContainer" containerID="0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd"
Nov 01 09:44:53.257267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.257235    4314 generic.go:356] "Generic (PLEG): container finished" podID="246bde58-8e09-43e3-b259-7097caf4a021" containerID="06a3b800afbd1f7c85dfbc791050801e06b07080f0a23c0f289c3d57d4ede483" exitCode=0
Nov 01 09:44:53.257418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.257306    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6" event={"ID":"246bde58-8e09-43e3-b259-7097caf4a021","Type":"ContainerDied","Data":"06a3b800afbd1f7c85dfbc791050801e06b07080f0a23c0f289c3d57d4ede483"}
Nov 01 09:44:53.267753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.267730    4314 scope.go:122] "RemoveContainer" containerID="0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd"
Nov 01 09:44:53.268135 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:44:53.268110    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd\": container with ID starting with 0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd not found: ID does not exist" containerID="0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd"
Nov 01 09:44:53.268249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.268145    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd"} err="failed to get container status \"0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd\": rpc error: code = NotFound desc = could not find container \"0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd\": container with ID starting with 0ee7adcdf0b84204a57c74d013c2b662b74c1adae0efd41b583f7ac332509fcd not found: ID does not exist"
Nov 01 09:44:53.292571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.292521    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:44:53.296397 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.296374    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume\") pod \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") "
Nov 01 09:44:53.296609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.296594    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb\" (UniqueName: \"kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb\") pod \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\" (UID: \"a53b33d4-51e4-497b-8b15-0dd05946ae8b\") "
Nov 01 09:44:53.296761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.296741    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume" pod "a53b33d4-51e4-497b-8b15-0dd05946ae8b" (UID: "a53b33d4-51e4-497b-8b15-0dd05946ae8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:44:53.297065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.297042    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53b33d4-51e4-497b-8b15-0dd05946ae8b-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:53.298577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.298559    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb" pod "a53b33d4-51e4-497b-8b15-0dd05946ae8b" (UID: "a53b33d4-51e4-497b-8b15-0dd05946ae8b"). InnerVolumeSpecName "kube-api-access-b6pnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:44:53.356835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.356798    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:53.398324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.398286    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume\") pod \"246bde58-8e09-43e3-b259-7097caf4a021\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") "
Nov 01 09:44:53.398524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.398448    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6\" (UniqueName: \"kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6\") pod \"246bde58-8e09-43e3-b259-7097caf4a021\" (UID: \"246bde58-8e09-43e3-b259-7097caf4a021\") "
Nov 01 09:44:53.398724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.398698    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume" pod "246bde58-8e09-43e3-b259-7097caf4a021" (UID: "246bde58-8e09-43e3-b259-7097caf4a021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:44:53.399026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.399007    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bde58-8e09-43e3-b259-7097caf4a021-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:53.399088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.399029    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pnb\" (UniqueName: \"kubernetes.io/projected/a53b33d4-51e4-497b-8b15-0dd05946ae8b-kube-api-access-b6pnb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:53.400575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.400552    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6" pod "246bde58-8e09-43e3-b259-7097caf4a021" (UID: "246bde58-8e09-43e3-b259-7097caf4a021"). InnerVolumeSpecName "kube-api-access-qn4l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:44:53.500937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.500894    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4l6\" (UniqueName: \"kubernetes.io/projected/246bde58-8e09-43e3-b259-7097caf4a021-kube-api-access-qn4l6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:44:53.670891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.670856    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"]
Nov 01 09:44:53.674758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:53.674731    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a53f2-58cc4dd74f-c278w"]
Nov 01 09:44:54.266888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.266862    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"
Nov 01 09:44:54.267373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.266861    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6" event={"ID":"246bde58-8e09-43e3-b259-7097caf4a021","Type":"ContainerDied","Data":"572e94c4e7c4a2b263877afafc392651d4ec8f6df2bc24d648cd4a4401203c77"}
Nov 01 09:44:54.267373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.266999    4314 scope.go:122] "RemoveContainer" containerID="06a3b800afbd1f7c85dfbc791050801e06b07080f0a23c0f289c3d57d4ede483"
Nov 01 09:44:54.420524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.420491    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"]
Nov 01 09:44:54.424801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.424774    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a53f2-f67f9f65f-xt4f6"]
Nov 01 09:44:54.779207 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.779178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:44:54.929966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.929937    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246bde58-8e09-43e3-b259-7097caf4a021" path="/var/lib/kubelet/pods/246bde58-8e09-43e3-b259-7097caf4a021/volumes"
Nov 01 09:44:54.930346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:54.930333    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53b33d4-51e4-497b-8b15-0dd05946ae8b" path="/var/lib/kubelet/pods/a53b33d4-51e4-497b-8b15-0dd05946ae8b/volumes"
Nov 01 09:44:58.105859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.105823    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"]
Nov 01 09:44:58.106431 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106417    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a53b33d4-51e4-497b-8b15-0dd05946ae8b" containerName="payload-processing"
Nov 01 09:44:58.106485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106432    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b33d4-51e4-497b-8b15-0dd05946ae8b" containerName="payload-processing"
Nov 01 09:44:58.106485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106453    4314 cpu_manager.go:414] "containerMap: removing container" podUID="246bde58-8e09-43e3-b259-7097caf4a021" containerName="payload-pre-processing"
Nov 01 09:44:58.106485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106460    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246bde58-8e09-43e3-b259-7097caf4a021" containerName="payload-pre-processing"
Nov 01 09:44:58.106485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106468    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f899b215-83e5-4a09-842a-9b641615f02d" containerName="revoke-keys"
Nov 01 09:44:58.106485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106473    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f899b215-83e5-4a09-842a-9b641615f02d" containerName="revoke-keys"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106498    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a27ae3d8-2666-4deb-87aa-e9b42aeba696" containerName="payload-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106503    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27ae3d8-2666-4deb-87aa-e9b42aeba696" containerName="payload-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106525    4314 cpu_manager.go:414] "containerMap: removing container" podUID="45e50de8-40d1-4fe6-8280-d0a6220657aa" containerName="payload-pre-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106543    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e50de8-40d1-4fe6-8280-d0a6220657aa" containerName="payload-pre-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106633    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="246bde58-8e09-43e3-b259-7097caf4a021" containerName="payload-pre-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106641    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f899b215-83e5-4a09-842a-9b641615f02d" containerName="revoke-keys"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106653    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27ae3d8-2666-4deb-87aa-e9b42aeba696" containerName="payload-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106661    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e50de8-40d1-4fe6-8280-d0a6220657aa" containerName="payload-pre-processing"
Nov 01 09:44:58.106687 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.106669    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b33d4-51e4-497b-8b15-0dd05946ae8b" containerName="payload-processing"
Nov 01 09:44:58.110506 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.110490    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.163474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.163441    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.163672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.163545    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mq9g\" (UniqueName: \"kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.163672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.163593    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.165359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.165334    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"]
Nov 01 09:44:58.206473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.206446    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:44:58.206473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.206455    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:44:58.206652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.206476    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:44:58.206652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.206484    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:44:58.264657 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.264629    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.264807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.264719    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mq9g\" (UniqueName: \"kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.264807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.264764    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.265388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.265353    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.267343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.267319    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.321480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.321449    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mq9g\" (UniqueName: \"kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g\") pod \"maas-api-revoke-sub-2f01411f-nqvqs\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.424159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.424096    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:44:58.795458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:58.795416    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"]
Nov 01 09:44:58.795686 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:44:58.795667    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa01d803_629f_4bd1_a5e8_afea9c1f417c.slice/crio-bf3eb80acc34d43cd4e73b039da06465f69285984e0732918b559fedaef9230e WatchSource:0}: Error finding container bf3eb80acc34d43cd4e73b039da06465f69285984e0732918b559fedaef9230e: Status 404 returned error can't find the container with id bf3eb80acc34d43cd4e73b039da06465f69285984e0732918b559fedaef9230e
Nov 01 09:44:59.308712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:59.308673    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs" event={"ID":"fa01d803-629f-4bd1-a5e8-afea9c1f417c","Type":"ContainerStarted","Data":"805290ea5354b09fe156ee6729c626f33d5f91471fc6c5cadf4deac442c93756"}
Nov 01 09:44:59.309143 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:59.308746    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs" event={"ID":"fa01d803-629f-4bd1-a5e8-afea9c1f417c","Type":"ContainerStarted","Data":"bf3eb80acc34d43cd4e73b039da06465f69285984e0732918b559fedaef9230e"}
Nov 01 09:44:59.418271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:59.418173    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs" podStartSLOduration=1.4181604669999999 podStartE2EDuration="1.418160467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:44:59.41742769 +0000 UTC m=+4372.611822246" watchObservedRunningTime="2026-11-01 09:44:59.418160467 +0000 UTC m=+4372.612555052"
Nov 01 09:44:59.984739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:44:59.984702    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:00.170684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.170652    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"]
Nov 01 09:45:00.174725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.174703    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"]
Nov 01 09:45:00.174889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.174813    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.178603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.178584    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"]
Nov 01 09:45:00.178725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.178684    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.182325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.182309    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"]
Nov 01 09:45:00.182444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.182423    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.187159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.187139    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.216696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.216674    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"]
Nov 01 09:45:00.220375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.220357    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"]
Nov 01 09:45:00.220480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.220460    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.224473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.224454    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"]
Nov 01 09:45:00.224594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.224553    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.238918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.237737    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"]
Nov 01 09:45:00.238918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.238879    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.245144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.245124    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"]
Nov 01 09:45:00.245262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.245232    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.250274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.250257    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"]
Nov 01 09:45:00.250386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.250370    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.256101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.255795    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"]
Nov 01 09:45:00.256101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.255897    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.273871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.266709    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"]
Nov 01 09:45:00.273871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.266748    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"]
Nov 01 09:45:00.273871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.269953    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.275861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.274256    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"]
Nov 01 09:45:00.276896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.276827    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.278688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.278672    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"]
Nov 01 09:45:00.278856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.278838    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.283394 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.283375    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"]
Nov 01 09:45:00.287023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.287002    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"]
Nov 01 09:45:00.290884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.290860    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.291011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.290920    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdknq\" (UniqueName: \"kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq\") pod \"maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.291011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.290996    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.291139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291022    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.291139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291043    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zvw\" (UniqueName: \"kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw\") pod \"maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.291139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291061    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lxz\" (UniqueName: \"kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.291311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291217    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.291311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291252    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pst7m\" (UniqueName: \"kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m\") pod \"maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.291311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291288    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"]
Nov 01 09:45:00.291472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291307    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.291472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291375    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl2m\" (UniqueName: \"kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m\") pod \"maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.291472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291446    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm\" (UID: \"45484203-f964-4732-a284-f501585412ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.291679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291485    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglgb\" (UniqueName: \"kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.291679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291579    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.291679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291618    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.291849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291728    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knppj\" (UniqueName: \"kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj\") pod \"maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.291849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291759    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.291849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291783    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.291849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291811    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blt2\" (UniqueName: \"kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.292073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291892    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6cq\" (UniqueName: \"kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.292073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.291919    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4z8\" (UniqueName: \"kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm\" (UID: \"45484203-f964-4732-a284-f501585412ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.294731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.294711    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"]
Nov 01 09:45:00.302147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.302129    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"]
Nov 01 09:45:00.302583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.302566    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.308290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.304507    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"]
Nov 01 09:45:00.328952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.328014    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"]
Nov 01 09:45:00.340054 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.338799    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"]
Nov 01 09:45:00.349591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.349513    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"]
Nov 01 09:45:00.350002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.349811    4314 generic.go:356] "Generic (PLEG): container finished" podID="fa01d803-629f-4bd1-a5e8-afea9c1f417c" containerID="805290ea5354b09fe156ee6729c626f33d5f91471fc6c5cadf4deac442c93756" exitCode=0
Nov 01 09:45:00.350002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.349872    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs" event={"ID":"fa01d803-629f-4bd1-a5e8-afea9c1f417c","Type":"ContainerDied","Data":"805290ea5354b09fe156ee6729c626f33d5f91471fc6c5cadf4deac442c93756"}
Nov 01 09:45:00.354350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.354330    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"]
Nov 01 09:45:00.374303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.372229    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"]
Nov 01 09:45:00.377115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.377086    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"]
Nov 01 09:45:00.393677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393647    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwz5k\" (UniqueName: \"kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k\") pod \"maas-api-key-cleanup-29892105-dlx9j\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.393798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393738    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9g9\" (UniqueName: \"kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.393857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393794    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.393931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393875    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdknq\" (UniqueName: \"kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq\") pod \"maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.393994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393927    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.393994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393959    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.393994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.393990    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zvw\" (UniqueName: \"kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw\") pod \"maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.394161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394017    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lxz\" (UniqueName: \"kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.394161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394083    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.394161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394110    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pst7m\" (UniqueName: \"kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m\") pod \"maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.394161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394139    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.394378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394167    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.394378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394203    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl2m\" (UniqueName: \"kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m\") pod \"maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.394378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394239    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.394378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394298    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm\" (UID: \"45484203-f964-4732-a284-f501585412ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.394378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394354    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglgb\" (UniqueName: \"kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394380    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9wh\" (UniqueName: \"kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh\") pod \"maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394430    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394461    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394489    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892105-dlx9j\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394572    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb29d\" (UniqueName: \"kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394626    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knppj\" (UniqueName: \"kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj\") pod \"maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394658    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394685    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.394728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394711    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blt2\" (UniqueName: \"kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.395019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394750    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6cq\" (UniqueName: \"kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.395019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394777    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4z8\" (UniqueName: \"kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm\" (UID: \"45484203-f964-4732-a284-f501585412ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.395019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.394826    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.395689 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.395667    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.396658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.396633    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.397311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.397284    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.397936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.397915    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.398055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.398030    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.405214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.398626    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.405214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.398741    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.405214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.399499    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm\" (UID: \"45484203-f964-4732-a284-f501585412ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.405214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.399558    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.405214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.399940    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.405214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.405196    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"]
Nov 01 09:45:00.454461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.454427    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lxz\" (UniqueName: \"kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.454461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.454451    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zvw\" (UniqueName: \"kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw\") pod \"maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.454746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.454544    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdknq\" (UniqueName: \"kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq\") pod \"maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.476293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.476246    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglgb\" (UniqueName: \"kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.476447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.476387    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4z8\" (UniqueName: \"kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm\" (UID: \"45484203-f964-4732-a284-f501585412ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.476447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.476405    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blt2\" (UniqueName: \"kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.476613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.476473    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl2m\" (UniqueName: \"kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m\") pod \"maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.476736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.476713    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6cq\" (UniqueName: \"kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.476963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.476945    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knppj\" (UniqueName: \"kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj\") pod \"maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.477778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.477749    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pst7m\" (UniqueName: \"kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m\") pod \"maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.490462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.490407    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:00.496231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496209    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.496324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496248    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.496324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496310    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9wh\" (UniqueName: \"kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh\") pod \"maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.496461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496343    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892105-dlx9j\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.496461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496396    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb29d\" (UniqueName: \"kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.496604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496470    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.496604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496546    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwz5k\" (UniqueName: \"kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k\") pod \"maas-api-key-cleanup-29892105-dlx9j\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.496725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496621    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9g9\" (UniqueName: \"kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.497007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.496986    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.497077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.497043    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.497136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.497116    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892105-dlx9j\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.497211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.497194    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.498829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.498807    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:00.505890 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.505868    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:00.515784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.515755    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:00.548574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.547863    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:00.559003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.558590    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:00.560355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.560260    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb29d\" (UniqueName: \"kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d\") pod \"maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.562258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.561897    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9wh\" (UniqueName: \"kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh\") pod \"maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.563059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.563022    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwz5k\" (UniqueName: \"kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k\") pod \"maas-api-key-cleanup-29892105-dlx9j\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.567200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.567156    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9g9\" (UniqueName: \"kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.597874    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:00.600040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.598291    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:00.652794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.652561    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:00.679776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.675896    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:00.679776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.676048    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:00.679776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.676108    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:00.687950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.687883    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:00.697390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.697358    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:00.893979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:00.893949    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"]
Nov 01 09:45:00.901456 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:00.901423    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73e0470_1f3a_4f32_999a_8527566eea4e.slice/crio-7b78a25faeedb088c3a81694ae61ae2ec927afda9981a523523b1e9b673c5537 WatchSource:0}: Error finding container 7b78a25faeedb088c3a81694ae61ae2ec927afda9981a523523b1e9b673c5537: Status 404 returned error can't find the container with id 7b78a25faeedb088c3a81694ae61ae2ec927afda9981a523523b1e9b673c5537
Nov 01 09:45:01.127006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.126716    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"]
Nov 01 09:45:01.131356 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.130316    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b34cb4_e295_466b_90b0_728ba394262f.slice/crio-ab7471b802d3aee7dac586e0941377f8dcde21f07904f3ba697bdb0e2c1de8e5 WatchSource:0}: Error finding container ab7471b802d3aee7dac586e0941377f8dcde21f07904f3ba697bdb0e2c1de8e5: Status 404 returned error can't find the container with id ab7471b802d3aee7dac586e0941377f8dcde21f07904f3ba697bdb0e2c1de8e5
Nov 01 09:45:01.131356 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.130717    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c4b756_4c9f_4cad_b807_063d3b9fe3ec.slice/crio-0fca520e9fee50691f1dea0b3cecfa7a719f098c72bcdd6b2db9ed724ea03142 WatchSource:0}: Error finding container 0fca520e9fee50691f1dea0b3cecfa7a719f098c72bcdd6b2db9ed724ea03142: Status 404 returned error can't find the container with id 0fca520e9fee50691f1dea0b3cecfa7a719f098c72bcdd6b2db9ed724ea03142
Nov 01 09:45:01.140482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.138898    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"]
Nov 01 09:45:01.172985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.172956    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"]
Nov 01 09:45:01.174068 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.174041    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb94c797_f3f1_449b_8a81_06a3640b45c2.slice/crio-2c841880cf6a4243fe602f0e1c50d364ed773948228fc6b47059d2b3a966d368 WatchSource:0}: Error finding container 2c841880cf6a4243fe602f0e1c50d364ed773948228fc6b47059d2b3a966d368: Status 404 returned error can't find the container with id 2c841880cf6a4243fe602f0e1c50d364ed773948228fc6b47059d2b3a966d368
Nov 01 09:45:01.178341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.178321    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"]
Nov 01 09:45:01.180017 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.179988    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefe2e59_a43a_4334_8ed2_6f5feca2c3b0.slice/crio-a4138ad25158d27217a07159051fe5f3aec0a7c52f56fc3398f29e661ac811c2 WatchSource:0}: Error finding container a4138ad25158d27217a07159051fe5f3aec0a7c52f56fc3398f29e661ac811c2: Status 404 returned error can't find the container with id a4138ad25158d27217a07159051fe5f3aec0a7c52f56fc3398f29e661ac811c2
Nov 01 09:45:01.238233 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.238210    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f53d4a4_30eb_46f7_b138_ce62135a0086.slice/crio-ad46add8f9a238cf47fbbb8c936540f38da8e04fe18b1079eb8ad96aa010dbe1 WatchSource:0}: Error finding container ad46add8f9a238cf47fbbb8c936540f38da8e04fe18b1079eb8ad96aa010dbe1: Status 404 returned error can't find the container with id ad46add8f9a238cf47fbbb8c936540f38da8e04fe18b1079eb8ad96aa010dbe1
Nov 01 09:45:01.238507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.238487    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"]
Nov 01 09:45:01.272159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.272085    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"]
Nov 01 09:45:01.275969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.275948    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"]
Nov 01 09:45:01.281162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.281142    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"]
Nov 01 09:45:01.359243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.359212    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerStarted","Data":"0fca520e9fee50691f1dea0b3cecfa7a719f098c72bcdd6b2db9ed724ea03142"}
Nov 01 09:45:01.360847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.360819    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerStarted","Data":"ab7471b802d3aee7dac586e0941377f8dcde21f07904f3ba697bdb0e2c1de8e5"}
Nov 01 09:45:01.362361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.362337    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerStarted","Data":"a4138ad25158d27217a07159051fe5f3aec0a7c52f56fc3398f29e661ac811c2"}
Nov 01 09:45:01.364770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.364737    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerStarted","Data":"ad46add8f9a238cf47fbbb8c936540f38da8e04fe18b1079eb8ad96aa010dbe1"}
Nov 01 09:45:01.367679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.367644    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerStarted","Data":"2c841880cf6a4243fe602f0e1c50d364ed773948228fc6b47059d2b3a966d368"}
Nov 01 09:45:01.369271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.369246    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"]
Nov 01 09:45:01.372386 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.371671    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34878064_c876_4a53_a449_0ba669063be3.slice/crio-ada7efbe47796b266299fdafe8c2b93b26269a6f3efbcae7a0f756c52ca24f1f WatchSource:0}: Error finding container ada7efbe47796b266299fdafe8c2b93b26269a6f3efbcae7a0f756c52ca24f1f: Status 404 returned error can't find the container with id ada7efbe47796b266299fdafe8c2b93b26269a6f3efbcae7a0f756c52ca24f1f
Nov 01 09:45:01.373760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.373598    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerStarted","Data":"7b78a25faeedb088c3a81694ae61ae2ec927afda9981a523523b1e9b673c5537"}
Nov 01 09:45:01.376461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.376304    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"]
Nov 01 09:45:01.379751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.379714    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"]
Nov 01 09:45:01.380093 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.380070    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c214a7_4c48_4bda_8fee_d44f737e394c.slice/crio-f5e02e7ef03a4d81bfb037f9827fbd45ff4a757a6bd7cb7e5c8c05c50ac7e87b WatchSource:0}: Error finding container f5e02e7ef03a4d81bfb037f9827fbd45ff4a757a6bd7cb7e5c8c05c50ac7e87b: Status 404 returned error can't find the container with id f5e02e7ef03a4d81bfb037f9827fbd45ff4a757a6bd7cb7e5c8c05c50ac7e87b
Nov 01 09:45:01.384201 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.384176    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaca18d_82ea_42ec_a1d9_22e21df4f1ab.slice/crio-a8e42e27e6813ebdf1e80f5aa7c26e246f99a2c280c19a11792b5e23b8327205 WatchSource:0}: Error finding container a8e42e27e6813ebdf1e80f5aa7c26e246f99a2c280c19a11792b5e23b8327205: Status 404 returned error can't find the container with id a8e42e27e6813ebdf1e80f5aa7c26e246f99a2c280c19a11792b5e23b8327205
Nov 01 09:45:01.426936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.426909    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"]
Nov 01 09:45:01.438678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.438656    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"]
Nov 01 09:45:01.577722 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:01.577669    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64975577_698a_42ea_b3d4_e27ebbdd46fb.slice/crio-02811a6560d8a533ebb5986f9a3f26ce708ab99e75c70ef31f3ff5576d622dc6 WatchSource:0}: Error finding container 02811a6560d8a533ebb5986f9a3f26ce708ab99e75c70ef31f3ff5576d622dc6: Status 404 returned error can't find the container with id 02811a6560d8a533ebb5986f9a3f26ce708ab99e75c70ef31f3ff5576d622dc6
Nov 01 09:45:01.784008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.783973    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:45:01.816854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.816829    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca\") pod \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") "
Nov 01 09:45:01.817052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.817038    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca\") pod \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") "
Nov 01 09:45:01.817182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.817161    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca" pod "fa01d803-629f-4bd1-a5e8-afea9c1f417c" (UID: "fa01d803-629f-4bd1-a5e8-afea9c1f417c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:01.817240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.817177    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g\" (UniqueName: \"kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g\") pod \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\" (UID: \"fa01d803-629f-4bd1-a5e8-afea9c1f417c\") "
Nov 01 09:45:01.817806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.817785    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:01.860598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.859821    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g" pod "fa01d803-629f-4bd1-a5e8-afea9c1f417c" (UID: "fa01d803-629f-4bd1-a5e8-afea9c1f417c"). InnerVolumeSpecName "kube-api-access-5mq9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:01.865125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.865089    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca" pod "fa01d803-629f-4bd1-a5e8-afea9c1f417c" (UID: "fa01d803-629f-4bd1-a5e8-afea9c1f417c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:45:01.918899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.918875    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa01d803-629f-4bd1-a5e8-afea9c1f417c-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:01.918974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.918900    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mq9g\" (UniqueName: \"kubernetes.io/projected/fa01d803-629f-4bd1-a5e8-afea9c1f417c-kube-api-access-5mq9g\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:01.923959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:01.923929    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:45:02.384947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.384917    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerStarted","Data":"3bb8bdfbd2a901d1155c2971a0af97d10307c7b7196aa86838036653ee2e0f90"}
Nov 01 09:45:02.386246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.386221    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerStarted","Data":"2c8f2b558813ada3cd338e9c61fd36ac6a2311683ceeb6a4aeb8f3708329bea2"}
Nov 01 09:45:02.387848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.387824    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerStarted","Data":"6ac881b35b29e591be45c45a2b228f4d47dd4753ff118c9820d291d32f929364"}
Nov 01 09:45:02.389361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.389340    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerStarted","Data":"d50b58fc91606b65759943e34f9a3c1c3852903f5c3c9da20a7f582c05e3419d"}
Nov 01 09:45:02.389451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.389375    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerStarted","Data":"ada7efbe47796b266299fdafe8c2b93b26269a6f3efbcae7a0f756c52ca24f1f"}
Nov 01 09:45:02.390691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.390676    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"
Nov 01 09:45:02.390691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.390679    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs" event={"ID":"fa01d803-629f-4bd1-a5e8-afea9c1f417c","Type":"ContainerDied","Data":"bf3eb80acc34d43cd4e73b039da06465f69285984e0732918b559fedaef9230e"}
Nov 01 09:45:02.390826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.390712    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf3eb80acc34d43cd4e73b039da06465f69285984e0732918b559fedaef9230e"
Nov 01 09:45:02.392049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.392025    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerStarted","Data":"9e29e02825d5ff5f42b4f1fdff19c4921a5014acd263f712bbba7ad4e22c3425"}
Nov 01 09:45:02.393602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.393579    4314 generic.go:356] "Generic (PLEG): container finished" podID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerID="5a6abb521e09fd672a3d18ff598115873870aed4584df58f8058632405d10067" exitCode=60
Nov 01 09:45:02.393693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.393662    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerDied","Data":"5a6abb521e09fd672a3d18ff598115873870aed4584df58f8058632405d10067"}
Nov 01 09:45:02.393856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.393838    4314 scope.go:122] "RemoveContainer" containerID="5a6abb521e09fd672a3d18ff598115873870aed4584df58f8058632405d10067"
Nov 01 09:45:02.395050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.395028    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerStarted","Data":"a8e42e27e6813ebdf1e80f5aa7c26e246f99a2c280c19a11792b5e23b8327205"}
Nov 01 09:45:02.396947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.396926    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" event={"ID":"b73ccde3-7970-49cc-8dd5-72b119cdcc47","Type":"ContainerStarted","Data":"5a8b0750883897dfb8e240c4214fa016dfc091d4980d2df07006bf8d3ca263eb"}
Nov 01 09:45:02.398368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.398334    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerStarted","Data":"2e26cb3c2aa39c3ca57889a1ecfc62396557f421f5cb4e71171ca7be44f18f65"}
Nov 01 09:45:02.400131 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.399964    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerStarted","Data":"02811a6560d8a533ebb5986f9a3f26ce708ab99e75c70ef31f3ff5576d622dc6"}
Nov 01 09:45:02.401773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.401754    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerStarted","Data":"f250c6a0c8f5d3f5d9ce9e0811e283dc8f7a035d9429631034a5d73b3ce48872"}
Nov 01 09:45:02.403302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.403278    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerStarted","Data":"f7fb95722fa65e07d2be582ab086ab5015157e302294ab5653f4a50674c53388"}
Nov 01 09:45:02.404756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.404737    4314 generic.go:356] "Generic (PLEG): container finished" podID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerID="a48879a683b6eed13da4ea69f2ba071e650772704fbd8c32f4445460cc185b6c" exitCode=60
Nov 01 09:45:02.404835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.404790    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerDied","Data":"a48879a683b6eed13da4ea69f2ba071e650772704fbd8c32f4445460cc185b6c"}
Nov 01 09:45:02.404991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.404978    4314 scope.go:122] "RemoveContainer" containerID="a48879a683b6eed13da4ea69f2ba071e650772704fbd8c32f4445460cc185b6c"
Nov 01 09:45:02.406009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.405987    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerStarted","Data":"f5e02e7ef03a4d81bfb037f9827fbd45ff4a757a6bd7cb7e5c8c05c50ac7e87b"}
Nov 01 09:45:02.494826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.494769    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" podStartSLOduration=2.494754112 podStartE2EDuration="2.494754112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.492198819 +0000 UTC m=+4375.686593372" watchObservedRunningTime="2026-11-01 09:45:02.494754112 +0000 UTC m=+4375.689148666"
Nov 01 09:45:02.606336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.606283    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" podStartSLOduration=2.606269007 podStartE2EDuration="2.606269007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.600588338 +0000 UTC m=+4375.794982894" watchObservedRunningTime="2026-11-01 09:45:02.606269007 +0000 UTC m=+4375.800663562"
Nov 01 09:45:02.710304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.710251    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" podStartSLOduration=2.710237879 podStartE2EDuration="2.710237879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.706681772 +0000 UTC m=+4375.901076327" watchObservedRunningTime="2026-11-01 09:45:02.710237879 +0000 UTC m=+4375.904632433"
Nov 01 09:45:02.817348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:02.817295    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" podStartSLOduration=2.817280578 podStartE2EDuration="2.817280578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.813062008 +0000 UTC m=+4376.007456567" watchObservedRunningTime="2026-11-01 09:45:02.817280578 +0000 UTC m=+4376.011675133"
Nov 01 09:45:03.027462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.027363    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" podStartSLOduration=3.027348945 podStartE2EDuration="3.027348945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.026662944 +0000 UTC m=+4376.221057497" watchObservedRunningTime="2026-11-01 09:45:03.027348945 +0000 UTC m=+4376.221743499"
Nov 01 09:45:03.240062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.240018    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" podStartSLOduration=3.240005699 podStartE2EDuration="3.240005699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.238675106 +0000 UTC m=+4376.433069661" watchObservedRunningTime="2026-11-01 09:45:03.240005699 +0000 UTC m=+4376.434400254"
Nov 01 09:45:03.347917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.347865    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" podStartSLOduration=3.3478503 podStartE2EDuration="3.3478503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.346960531 +0000 UTC m=+4376.541355088" watchObservedRunningTime="2026-11-01 09:45:03.3478503 +0000 UTC m=+4376.542244855"
Nov 01 09:45:03.415456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.415426    4314 generic.go:356] "Generic (PLEG): container finished" podID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerID="65b11a6d9817134c1f2e04d84cef60c679b5937e87fe57f0585a7001ec3ccbca" exitCode=60
Nov 01 09:45:03.415905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.415508    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerDied","Data":"65b11a6d9817134c1f2e04d84cef60c679b5937e87fe57f0585a7001ec3ccbca"}
Nov 01 09:45:03.415905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.415819    4314 scope.go:122] "RemoveContainer" containerID="65b11a6d9817134c1f2e04d84cef60c679b5937e87fe57f0585a7001ec3ccbca"
Nov 01 09:45:03.417305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.417281    4314 generic.go:356] "Generic (PLEG): container finished" podID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerID="f250c6a0c8f5d3f5d9ce9e0811e283dc8f7a035d9429631034a5d73b3ce48872" exitCode=60
Nov 01 09:45:03.417416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.417362    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerDied","Data":"f250c6a0c8f5d3f5d9ce9e0811e283dc8f7a035d9429631034a5d73b3ce48872"}
Nov 01 09:45:03.417579 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.417560    4314 scope.go:122] "RemoveContainer" containerID="f250c6a0c8f5d3f5d9ce9e0811e283dc8f7a035d9429631034a5d73b3ce48872"
Nov 01 09:45:03.419292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.419262    4314 generic.go:356] "Generic (PLEG): container finished" podID="25b34cb4-e295-466b-90b0-728ba394262f" containerID="f7fb95722fa65e07d2be582ab086ab5015157e302294ab5653f4a50674c53388" exitCode=60
Nov 01 09:45:03.419353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.419333    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerDied","Data":"f7fb95722fa65e07d2be582ab086ab5015157e302294ab5653f4a50674c53388"}
Nov 01 09:45:03.419630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.419592    4314 scope.go:122] "RemoveContainer" containerID="f7fb95722fa65e07d2be582ab086ab5015157e302294ab5653f4a50674c53388"
Nov 01 09:45:03.421492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.421467    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerStarted","Data":"2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d"}
Nov 01 09:45:03.423065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.423046    4314 generic.go:356] "Generic (PLEG): container finished" podID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerID="da61d92460d3f326c1a77c22e67acdcbfd7083349dfa66ae04ad90c44f3831b5" exitCode=60
Nov 01 09:45:03.423145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.423128    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerDied","Data":"da61d92460d3f326c1a77c22e67acdcbfd7083349dfa66ae04ad90c44f3831b5"}
Nov 01 09:45:03.423305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.423290    4314 scope.go:122] "RemoveContainer" containerID="da61d92460d3f326c1a77c22e67acdcbfd7083349dfa66ae04ad90c44f3831b5"
Nov 01 09:45:03.424826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.424806    4314 generic.go:356] "Generic (PLEG): container finished" podID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerID="3bb8bdfbd2a901d1155c2971a0af97d10307c7b7196aa86838036653ee2e0f90" exitCode=60
Nov 01 09:45:03.424934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.424874    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerDied","Data":"3bb8bdfbd2a901d1155c2971a0af97d10307c7b7196aa86838036653ee2e0f90"}
Nov 01 09:45:03.425159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.425145    4314 scope.go:122] "RemoveContainer" containerID="3bb8bdfbd2a901d1155c2971a0af97d10307c7b7196aa86838036653ee2e0f90"
Nov 01 09:45:03.426376 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.426353    4314 generic.go:356] "Generic (PLEG): container finished" podID="45484203-f964-4732-a284-f501585412ac" containerID="0799c71ebd688821764eeb64c6b808e4d4f2a362c0ac27475059fb1451376db3" exitCode=60
Nov 01 09:45:03.426477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.426431    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerDied","Data":"0799c71ebd688821764eeb64c6b808e4d4f2a362c0ac27475059fb1451376db3"}
Nov 01 09:45:03.426626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.426611    4314 scope.go:122] "RemoveContainer" containerID="0799c71ebd688821764eeb64c6b808e4d4f2a362c0ac27475059fb1451376db3"
Nov 01 09:45:03.428459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.428439    4314 generic.go:356] "Generic (PLEG): container finished" podID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerID="6ac881b35b29e591be45c45a2b228f4d47dd4753ff118c9820d291d32f929364" exitCode=60
Nov 01 09:45:03.428572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.428513    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerDied","Data":"6ac881b35b29e591be45c45a2b228f4d47dd4753ff118c9820d291d32f929364"}
Nov 01 09:45:03.428906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.428885    4314 scope.go:122] "RemoveContainer" containerID="6ac881b35b29e591be45c45a2b228f4d47dd4753ff118c9820d291d32f929364"
Nov 01 09:45:03.430059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.430038    4314 generic.go:356] "Generic (PLEG): container finished" podID="34878064-c876-4a53-a449-0ba669063be3" containerID="d50b58fc91606b65759943e34f9a3c1c3852903f5c3c9da20a7f582c05e3419d" exitCode=60
Nov 01 09:45:03.430175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.430114    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerDied","Data":"d50b58fc91606b65759943e34f9a3c1c3852903f5c3c9da20a7f582c05e3419d"}
Nov 01 09:45:03.430292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.430277    4314 scope.go:122] "RemoveContainer" containerID="d50b58fc91606b65759943e34f9a3c1c3852903f5c3c9da20a7f582c05e3419d"
Nov 01 09:45:03.431608 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.431590    4314 generic.go:356] "Generic (PLEG): container finished" podID="100116a4-7b5a-4083-8d89-ab55162f065b" containerID="4125106d778feae10bd70fd12159ec98d48cce7a5182c391d381fa26d5a90849" exitCode=60
Nov 01 09:45:03.431711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.431661    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerDied","Data":"4125106d778feae10bd70fd12159ec98d48cce7a5182c391d381fa26d5a90849"}
Nov 01 09:45:03.431866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.431849    4314 scope.go:122] "RemoveContainer" containerID="4125106d778feae10bd70fd12159ec98d48cce7a5182c391d381fa26d5a90849"
Nov 01 09:45:03.433443 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.433420    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerStarted","Data":"dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87"}
Nov 01 09:45:03.436996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.436954    4314 generic.go:356] "Generic (PLEG): container finished" podID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerID="4b61fefae0e7e391bd07750844bc685d8597988fd42f23eb8460938cb87b62cd" exitCode=60
Nov 01 09:45:03.437323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.437286    4314 scope.go:122] "RemoveContainer" containerID="4b61fefae0e7e391bd07750844bc685d8597988fd42f23eb8460938cb87b62cd"
Nov 01 09:45:03.437637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.437613    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerDied","Data":"4b61fefae0e7e391bd07750844bc685d8597988fd42f23eb8460938cb87b62cd"}
Nov 01 09:45:03.440046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.440027    4314 generic.go:356] "Generic (PLEG): container finished" podID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerID="7e907169755308b21938e1dc57d87ec14f00ee1d28a428ffeaaad3e51fb1190b" exitCode=60
Nov 01 09:45:03.440146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.440102    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" event={"ID":"b73ccde3-7970-49cc-8dd5-72b119cdcc47","Type":"ContainerDied","Data":"7e907169755308b21938e1dc57d87ec14f00ee1d28a428ffeaaad3e51fb1190b"}
Nov 01 09:45:03.440323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.440308    4314 scope.go:122] "RemoveContainer" containerID="7e907169755308b21938e1dc57d87ec14f00ee1d28a428ffeaaad3e51fb1190b"
Nov 01 09:45:03.443101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.443082    4314 generic.go:356] "Generic (PLEG): container finished" podID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerID="08b03708cdb2151db6d8bf550fc62c9084fe5b0f0ca05fe24e89fb45380cfa37" exitCode=60
Nov 01 09:45:03.443190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.443150    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerDied","Data":"08b03708cdb2151db6d8bf550fc62c9084fe5b0f0ca05fe24e89fb45380cfa37"}
Nov 01 09:45:03.443369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.443352    4314 scope.go:122] "RemoveContainer" containerID="08b03708cdb2151db6d8bf550fc62c9084fe5b0f0ca05fe24e89fb45380cfa37"
Nov 01 09:45:03.959071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:03.959019    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" podStartSLOduration=3.959004861 podStartE2EDuration="3.959004861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.955875839 +0000 UTC m=+4377.150270395" watchObservedRunningTime="2026-11-01 09:45:03.959004861 +0000 UTC m=+4377.153399437"
Nov 01 09:45:04.175292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.174813    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" podStartSLOduration=4.174795042 podStartE2EDuration="4.174795042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:04.169465662 +0000 UTC m=+4377.363860228" watchObservedRunningTime="2026-11-01 09:45:04.174795042 +0000 UTC m=+4377.369189598"
Nov 01 09:45:04.452653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.452566    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerStarted","Data":"c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005"}
Nov 01 09:45:04.454547 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.454510    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerStarted","Data":"8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05"}
Nov 01 09:45:04.456349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.456322    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerStarted","Data":"57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da"}
Nov 01 09:45:04.464389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.464360    4314 generic.go:356] "Generic (PLEG): container finished" podID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerID="2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d" exitCode=60
Nov 01 09:45:04.464771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.464459    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerDied","Data":"2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d"}
Nov 01 09:45:04.464771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.464516    4314 scope.go:122] "RemoveContainer" containerID="a48879a683b6eed13da4ea69f2ba071e650772704fbd8c32f4445460cc185b6c"
Nov 01 09:45:04.464771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.464673    4314 scope.go:122] "RemoveContainer" containerID="2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d"
Nov 01 09:45:04.464977 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:04.464954    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x_odh-ai-gateway-infra(fefe2e59-a43a-4334-8ed2-6f5feca2c3b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0"
Nov 01 09:45:04.478377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.478347    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerStarted","Data":"75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7"}
Nov 01 09:45:04.481619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.481593    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerStarted","Data":"330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb"}
Nov 01 09:45:04.483862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.483835    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerStarted","Data":"97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4"}
Nov 01 09:45:04.485924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.485899    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerStarted","Data":"1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2"}
Nov 01 09:45:04.487825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.487804    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerStarted","Data":"127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec"}
Nov 01 09:45:04.490135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.490108    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerStarted","Data":"b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22"}
Nov 01 09:45:04.492059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.492040    4314 generic.go:356] "Generic (PLEG): container finished" podID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerID="dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87" exitCode=60
Nov 01 09:45:04.492151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.492106    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerDied","Data":"dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87"}
Nov 01 09:45:04.492196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.492150    4314 scope.go:122] "RemoveContainer" containerID="5a6abb521e09fd672a3d18ff598115873870aed4584df58f8058632405d10067"
Nov 01 09:45:04.492368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.492349    4314 scope.go:122] "RemoveContainer" containerID="dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87"
Nov 01 09:45:04.492675 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:04.492649    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7_odh-ai-gateway-infra(e73e0470-1f3a-4f32-999a-8527566eea4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e"
Nov 01 09:45:04.494067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.494047    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerStarted","Data":"fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8"}
Nov 01 09:45:04.496036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.496018    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" event={"ID":"b73ccde3-7970-49cc-8dd5-72b119cdcc47","Type":"ContainerStarted","Data":"7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33"}
Nov 01 09:45:04.497923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.497903    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerStarted","Data":"265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c"}
Nov 01 09:45:04.792435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:04.792406    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:05.131517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.131472    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" podStartSLOduration=5.131458921 podStartE2EDuration="5.131458921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:05.130145559 +0000 UTC m=+4378.324540125" watchObservedRunningTime="2026-11-01 09:45:05.131458921 +0000 UTC m=+4378.325853475"
Nov 01 09:45:05.455614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.455566    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" podStartSLOduration=5.455553145 podStartE2EDuration="5.455553145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:05.453926427 +0000 UTC m=+4378.648320981" watchObservedRunningTime="2026-11-01 09:45:05.455553145 +0000 UTC m=+4378.649947696"
Nov 01 09:45:05.507696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.507662    4314 generic.go:356] "Generic (PLEG): container finished" podID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerID="330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb" exitCode=60
Nov 01 09:45:05.507875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.507751    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerDied","Data":"330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb"}
Nov 01 09:45:05.507875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.507825    4314 scope.go:122] "RemoveContainer" containerID="3bb8bdfbd2a901d1155c2971a0af97d10307c7b7196aa86838036653ee2e0f90"
Nov 01 09:45:05.508079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.508060    4314 scope.go:122] "RemoveContainer" containerID="330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb"
Nov 01 09:45:05.508446 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.508420    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr_odh-ai-gateway-infra(3f53d4a4-30eb-46f7-b138-ce62135a0086)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086"
Nov 01 09:45:05.509854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.509833    4314 generic.go:356] "Generic (PLEG): container finished" podID="45484203-f964-4732-a284-f501585412ac" containerID="97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4" exitCode=60
Nov 01 09:45:05.509977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.509910    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerDied","Data":"97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4"}
Nov 01 09:45:05.510142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.510130    4314 scope.go:122] "RemoveContainer" containerID="97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4"
Nov 01 09:45:05.510438 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.510414    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm_odh-ai-gateway-infra(45484203-f964-4732-a284-f501585412ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" podUID="45484203-f964-4732-a284-f501585412ac"
Nov 01 09:45:05.511770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.511750    4314 generic.go:356] "Generic (PLEG): container finished" podID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerID="1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2" exitCode=60
Nov 01 09:45:05.511865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.511814    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerDied","Data":"1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2"}
Nov 01 09:45:05.512053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.512039    4314 scope.go:122] "RemoveContainer" containerID="1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2"
Nov 01 09:45:05.512347 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.512326    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx_odh-ai-gateway-infra(eb94c797-f3f1-449b-8a81-06a3640b45c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2"
Nov 01 09:45:05.513877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.513858    4314 generic.go:356] "Generic (PLEG): container finished" podID="34878064-c876-4a53-a449-0ba669063be3" containerID="127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec" exitCode=60
Nov 01 09:45:05.513972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.513939    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerDied","Data":"127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec"}
Nov 01 09:45:05.514125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.514112    4314 scope.go:122] "RemoveContainer" containerID="127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec"
Nov 01 09:45:05.514361 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.514334    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x_odh-ai-gateway-infra(34878064-c876-4a53-a449-0ba669063be3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" podUID="34878064-c876-4a53-a449-0ba669063be3"
Nov 01 09:45:05.515945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.515926    4314 generic.go:356] "Generic (PLEG): container finished" podID="100116a4-7b5a-4083-8d89-ab55162f065b" containerID="b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22" exitCode=60
Nov 01 09:45:05.516033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.516013    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerDied","Data":"b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22"}
Nov 01 09:45:05.516258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.516239    4314 scope.go:122] "RemoveContainer" containerID="b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22"
Nov 01 09:45:05.516518 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.516495    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7_odh-ai-gateway-infra(100116a4-7b5a-4083-8d89-ab55162f065b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" podUID="100116a4-7b5a-4083-8d89-ab55162f065b"
Nov 01 09:45:05.518107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.518090    4314 scope.go:122] "RemoveContainer" containerID="dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87"
Nov 01 09:45:05.518332 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.518315    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7_odh-ai-gateway-infra(e73e0470-1f3a-4f32-999a-8527566eea4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e"
Nov 01 09:45:05.519715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.519694    4314 generic.go:356] "Generic (PLEG): container finished" podID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerID="fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8" exitCode=60
Nov 01 09:45:05.519808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.519761    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerDied","Data":"fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8"}
Nov 01 09:45:05.519967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.519951    4314 scope.go:122] "RemoveContainer" containerID="fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8"
Nov 01 09:45:05.520188 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.520167    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg_odh-ai-gateway-infra(cfaca18d-82ea-42ec-a1d9-22e21df4f1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab"
Nov 01 09:45:05.521628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.521610    4314 generic.go:356] "Generic (PLEG): container finished" podID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerID="7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33" exitCode=60
Nov 01 09:45:05.521719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.521633    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" event={"ID":"b73ccde3-7970-49cc-8dd5-72b119cdcc47","Type":"ContainerDied","Data":"7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33"}
Nov 01 09:45:05.521885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.521869    4314 scope.go:122] "RemoveContainer" containerID="7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33"
Nov 01 09:45:05.522095 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.522074    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd_odh-ai-gateway-infra(b73ccde3-7970-49cc-8dd5-72b119cdcc47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47"
Nov 01 09:45:05.522840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.522820    4314 scope.go:122] "RemoveContainer" containerID="0799c71ebd688821764eeb64c6b808e4d4f2a362c0ac27475059fb1451376db3"
Nov 01 09:45:05.523639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.523542    4314 generic.go:356] "Generic (PLEG): container finished" podID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerID="265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c" exitCode=60
Nov 01 09:45:05.523639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.523557    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerDied","Data":"265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c"}
Nov 01 09:45:05.523842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.523825    4314 scope.go:122] "RemoveContainer" containerID="265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c"
Nov 01 09:45:05.524050 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.524032    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892105-dlx9j_odh-ai-gateway-infra(e7f9c43d-f941-45f7-b20f-635cd0a83cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5"
Nov 01 09:45:05.525458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.525438    4314 generic.go:356] "Generic (PLEG): container finished" podID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerID="c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005" exitCode=60
Nov 01 09:45:05.525585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.525521    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerDied","Data":"c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005"}
Nov 01 09:45:05.525715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.525700    4314 scope.go:122] "RemoveContainer" containerID="c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005"
Nov 01 09:45:05.525889 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.525870    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc_odh-ai-gateway-infra(64975577-698a-42ea-b3d4-e27ebbdd46fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb"
Nov 01 09:45:05.527276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.527258    4314 generic.go:356] "Generic (PLEG): container finished" podID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerID="8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05" exitCode=60
Nov 01 09:45:05.527361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.527311    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerDied","Data":"8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05"}
Nov 01 09:45:05.527495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.527477    4314 scope.go:122] "RemoveContainer" containerID="8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05"
Nov 01 09:45:05.527707 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.527682    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz_odh-ai-gateway-infra(c7c4b756-4c9f-4cad-b807-063d3b9fe3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec"
Nov 01 09:45:05.529201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.529183    4314 generic.go:356] "Generic (PLEG): container finished" podID="25b34cb4-e295-466b-90b0-728ba394262f" containerID="57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da" exitCode=60
Nov 01 09:45:05.529296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.529249    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerDied","Data":"57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da"}
Nov 01 09:45:05.529475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.529460    4314 scope.go:122] "RemoveContainer" containerID="57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da"
Nov 01 09:45:05.529831 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.529804    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77_odh-ai-gateway-infra(25b34cb4-e295-466b-90b0-728ba394262f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" podUID="25b34cb4-e295-466b-90b0-728ba394262f"
Nov 01 09:45:05.531369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.531350    4314 scope.go:122] "RemoveContainer" containerID="2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d"
Nov 01 09:45:05.531641 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.531616    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x_odh-ai-gateway-infra(fefe2e59-a43a-4334-8ed2-6f5feca2c3b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0"
Nov 01 09:45:05.533462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.533438    4314 generic.go:356] "Generic (PLEG): container finished" podID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerID="75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7" exitCode=60
Nov 01 09:45:05.533586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.533496    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerDied","Data":"75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7"}
Nov 01 09:45:05.533799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.533779    4314 scope.go:122] "RemoveContainer" containerID="75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7"
Nov 01 09:45:05.534069 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:05.534044    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn_odh-ai-gateway-infra(34c214a7-4c48-4bda-8fee-d44f737e394c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c"
Nov 01 09:45:05.539374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.539351    4314 scope.go:122] "RemoveContainer" containerID="6ac881b35b29e591be45c45a2b228f4d47dd4753ff118c9820d291d32f929364"
Nov 01 09:45:05.552255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.552238    4314 scope.go:122] "RemoveContainer" containerID="d50b58fc91606b65759943e34f9a3c1c3852903f5c3c9da20a7f582c05e3419d"
Nov 01 09:45:05.565654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.565630    4314 scope.go:122] "RemoveContainer" containerID="4125106d778feae10bd70fd12159ec98d48cce7a5182c391d381fa26d5a90849"
Nov 01 09:45:05.581861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.580204    4314 scope.go:122] "RemoveContainer" containerID="4b61fefae0e7e391bd07750844bc685d8597988fd42f23eb8460938cb87b62cd"
Nov 01 09:45:05.594526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.594501    4314 scope.go:122] "RemoveContainer" containerID="7e907169755308b21938e1dc57d87ec14f00ee1d28a428ffeaaad3e51fb1190b"
Nov 01 09:45:05.613228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.613195    4314 scope.go:122] "RemoveContainer" containerID="08b03708cdb2151db6d8bf550fc62c9084fe5b0f0ca05fe24e89fb45380cfa37"
Nov 01 09:45:05.625681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.625663    4314 scope.go:122] "RemoveContainer" containerID="65b11a6d9817134c1f2e04d84cef60c679b5937e87fe57f0585a7001ec3ccbca"
Nov 01 09:45:05.639863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.639844    4314 scope.go:122] "RemoveContainer" containerID="f250c6a0c8f5d3f5d9ce9e0811e283dc8f7a035d9429631034a5d73b3ce48872"
Nov 01 09:45:05.652125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.652106    4314 scope.go:122] "RemoveContainer" containerID="f7fb95722fa65e07d2be582ab086ab5015157e302294ab5653f4a50674c53388"
Nov 01 09:45:05.664952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.664931    4314 scope.go:122] "RemoveContainer" containerID="da61d92460d3f326c1a77c22e67acdcbfd7083349dfa66ae04ad90c44f3831b5"
Nov 01 09:45:05.668975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.668936    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" podStartSLOduration=5.668924606 podStartE2EDuration="5.668924606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:05.668011907 +0000 UTC m=+4378.862406461" watchObservedRunningTime="2026-11-01 09:45:05.668924606 +0000 UTC m=+4378.863319160"
Nov 01 09:45:05.778400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.778339    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" podStartSLOduration=5.778325508 podStartE2EDuration="5.778325508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:05.775892318 +0000 UTC m=+4378.970286875" watchObservedRunningTime="2026-11-01 09:45:05.778325508 +0000 UTC m=+4378.972720062"
Nov 01 09:45:05.990685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:05.990636    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" podStartSLOduration=5.990622679 podStartE2EDuration="5.990622679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:05.988899454 +0000 UTC m=+4379.183294013" watchObservedRunningTime="2026-11-01 09:45:05.990622679 +0000 UTC m=+4379.185017234"
Nov 01 09:45:06.544452 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.544428    4314 scope.go:122] "RemoveContainer" containerID="c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005"
Nov 01 09:45:06.544947 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.544782    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc_odh-ai-gateway-infra(64975577-698a-42ea-b3d4-e27ebbdd46fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb"
Nov 01 09:45:06.546241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.546219    4314 scope.go:122] "RemoveContainer" containerID="8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05"
Nov 01 09:45:06.546494 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.546467    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz_odh-ai-gateway-infra(c7c4b756-4c9f-4cad-b807-063d3b9fe3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec"
Nov 01 09:45:06.548020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.547997    4314 scope.go:122] "RemoveContainer" containerID="57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da"
Nov 01 09:45:06.548234 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.548214    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77_odh-ai-gateway-infra(25b34cb4-e295-466b-90b0-728ba394262f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" podUID="25b34cb4-e295-466b-90b0-728ba394262f"
Nov 01 09:45:06.549642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.549624    4314 scope.go:122] "RemoveContainer" containerID="75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7"
Nov 01 09:45:06.549861 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.549839    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn_odh-ai-gateway-infra(34c214a7-4c48-4bda-8fee-d44f737e394c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c"
Nov 01 09:45:06.551177 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.551156    4314 scope.go:122] "RemoveContainer" containerID="330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb"
Nov 01 09:45:06.551394 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.551373    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr_odh-ai-gateway-infra(3f53d4a4-30eb-46f7-b138-ce62135a0086)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086"
Nov 01 09:45:06.552638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.552622    4314 scope.go:122] "RemoveContainer" containerID="97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4"
Nov 01 09:45:06.552832 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.552815    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm_odh-ai-gateway-infra(45484203-f964-4732-a284-f501585412ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" podUID="45484203-f964-4732-a284-f501585412ac"
Nov 01 09:45:06.554093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.554076    4314 scope.go:122] "RemoveContainer" containerID="1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2"
Nov 01 09:45:06.554305 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.554284    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx_odh-ai-gateway-infra(eb94c797-f3f1-449b-8a81-06a3640b45c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2"
Nov 01 09:45:06.555656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.555639    4314 scope.go:122] "RemoveContainer" containerID="127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec"
Nov 01 09:45:06.555854 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.555835    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x_odh-ai-gateway-infra(34878064-c876-4a53-a449-0ba669063be3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" podUID="34878064-c876-4a53-a449-0ba669063be3"
Nov 01 09:45:06.557200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.557182    4314 scope.go:122] "RemoveContainer" containerID="b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22"
Nov 01 09:45:06.557394 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.557375    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7_odh-ai-gateway-infra(100116a4-7b5a-4083-8d89-ab55162f065b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" podUID="100116a4-7b5a-4083-8d89-ab55162f065b"
Nov 01 09:45:06.558677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.558659    4314 scope.go:122] "RemoveContainer" containerID="fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8"
Nov 01 09:45:06.558899 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.558879    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg_odh-ai-gateway-infra(cfaca18d-82ea-42ec-a1d9-22e21df4f1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab"
Nov 01 09:45:06.560242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.560224    4314 scope.go:122] "RemoveContainer" containerID="7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33"
Nov 01 09:45:06.560464 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.560442    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd_odh-ai-gateway-infra(b73ccde3-7970-49cc-8dd5-72b119cdcc47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47"
Nov 01 09:45:06.561720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:06.561704    4314 scope.go:122] "RemoveContainer" containerID="265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c"
Nov 01 09:45:06.561891 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:06.561876    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892105-dlx9j_odh-ai-gateway-infra(e7f9c43d-f941-45f7-b20f-635cd0a83cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5"
Nov 01 09:45:09.713942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:09.713874    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:10.556525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:10.555840    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"]
Nov 01 09:45:10.594439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:10.594408    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"]
Nov 01 09:45:10.595613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:10.595587    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:45:10.599646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:10.599625    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:10.760584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:10.760551    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:11.417784 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:11.417742    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:45:12.707714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:12.707590    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:13.200154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.200117    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"]
Nov 01 09:45:13.201733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.201707    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fa01d803-629f-4bd1-a5e8-afea9c1f417c" containerName="revoke-keys"
Nov 01 09:45:13.201908 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.201892    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa01d803-629f-4bd1-a5e8-afea9c1f417c" containerName="revoke-keys"
Nov 01 09:45:13.203467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.203446    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa01d803-629f-4bd1-a5e8-afea9c1f417c" containerName="revoke-keys"
Nov 01 09:45:13.209830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.209811    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.327549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.321239    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3710de-openshift-default-dockercfg-6g8wf\""
Nov 01 09:45:13.335566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.329957    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.336711    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.336784    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.336841    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.336891    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlh5\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.336931    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.337084    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.337155    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.337222    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.337590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.337298    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.342997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.342974    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"]
Nov 01 09:45:13.438863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.438815    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.439076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.438895    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.439076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.438939    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlh5\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.479833 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.479742    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.480387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.480352    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.480811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.480790    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.481026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.481009    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.481524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.481499    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.481881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.481803    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.482912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.482879    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.484055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.484033    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.484463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.484441    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.484893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.484869    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.485243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.485222    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.495636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.495611    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.516147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.516090    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:13.586555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.585767    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlh5\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.587805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.587774    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.589651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.589595    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token\") pod \"e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") " pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:13.791073 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:13.791024    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0297be67dfdc: secret "agc-cred-0297be67dfdc" not found
Nov 01 09:45:13.791073 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:13.791062    4314 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-47f306d7be56: secret "agc-cred-47f306d7be56" not found
Nov 01 09:45:13.791073 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:13.791079    4314 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr: [secret "agc-cred-0297be67dfdc" not found, secret "agc-cred-47f306d7be56" not found]
Nov 01 09:45:13.791820 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:13.791144    4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials podName:8c4063bd-75d1-44e0-8885-1dab686a9f29 nodeName:}" failed. No retries permitted until 2026-11-01 09:47:15.791124428 +0000 UTC m=+4508.985518980 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials") pod "payload-processing-external-model-fd5d559f7-gmknr" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29") : [secret "agc-cred-0297be67dfdc" not found, secret "agc-cred-47f306d7be56" not found]
Nov 01 09:45:13.835404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:13.835361    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:14.318751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.318704    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:14.318901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.318803    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:14.318901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.318849    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:14.325195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.322445    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"]
Nov 01 09:45:14.396365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.396305    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"]
Nov 01 09:45:14.397970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.397101    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy" containerID="cri-o://13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e" gracePeriod=30
Nov 01 09:45:14.409230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.408839    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"]
Nov 01 09:45:14.409230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.409072    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy" containerID="cri-o://ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c" gracePeriod=30
Nov 01 09:45:14.428208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.428085    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"]
Nov 01 09:45:14.428951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.428882    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy" containerID="cri-o://b7e15eeed0a0e40b87276e76718c3384c1cd917c84282175dfb0b426808c6240" gracePeriod=30
Nov 01 09:45:14.433600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.433574    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"]
Nov 01 09:45:14.433768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.433749    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy" containerID="cri-o://272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066" gracePeriod=30
Nov 01 09:45:14.640396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.640311    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" event={"ID":"0ada1f59-3446-416f-900c-4e7a192b4399","Type":"ContainerStarted","Data":"abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"}
Nov 01 09:45:14.640396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.640364    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" event={"ID":"0ada1f59-3446-416f-900c-4e7a192b4399","Type":"ContainerStarted","Data":"88f88006535833a67bcc9f26acfbf0e380283b45078c64d2f826d4bdfc650b69"}
Nov 01 09:45:14.720062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.720028    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:14.845675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.845642    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:14.998291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:14.998232    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podStartSLOduration=1.9982144339999999 podStartE2EDuration="1.998214434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:14.994658399 +0000 UTC m=+4388.189052955" watchObservedRunningTime="2026-11-01 09:45:14.998214434 +0000 UTC m=+4388.192609062"
Nov 01 09:45:15.556726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.556672    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr" podUID="8c4063bd-75d1-44e0-8885-1dab686a9f29" containerName="payload-processing" containerID="cri-o://03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e" gracePeriod=25
Nov 01 09:45:15.595722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.595690    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing" containerID="cri-o://606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7" gracePeriod=25
Nov 01 09:45:15.849729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.849385    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:15.896841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.896813    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:45:15.915116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.915094    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:45:15.921777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.921755    4314 scope.go:122] "RemoveContainer" containerID="2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d"
Nov 01 09:45:15.921944 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.921924    4314 scope.go:122] "RemoveContainer" containerID="b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22"
Nov 01 09:45:15.922027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.922011    4314 scope.go:122] "RemoveContainer" containerID="97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4"
Nov 01 09:45:15.922136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.922118    4314 scope.go:122] "RemoveContainer" containerID="330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb"
Nov 01 09:45:15.922215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.922198    4314 scope.go:122] "RemoveContainer" containerID="265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c"
Nov 01 09:45:15.922302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.922283    4314 scope.go:122] "RemoveContainer" containerID="dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87"
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978167    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials\") pod \"cbb7ed3b-4486-44b8-8842-8c857457a124\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978212    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb\") pod \"8c4063bd-75d1-44e0-8885-1dab686a9f29\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978269    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials\") pod \"8c4063bd-75d1-44e0-8885-1dab686a9f29\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978360    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume\") pod \"cbb7ed3b-4486-44b8-8842-8c857457a124\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978492    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s\") pod \"cbb7ed3b-4486-44b8-8842-8c857457a124\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978581    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume\") pod \"cbb7ed3b-4486-44b8-8842-8c857457a124\" (UID: \"cbb7ed3b-4486-44b8-8842-8c857457a124\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978620    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume\") pod \"8c4063bd-75d1-44e0-8885-1dab686a9f29\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.978680    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume\") pod \"8c4063bd-75d1-44e0-8885-1dab686a9f29\" (UID: \"8c4063bd-75d1-44e0-8885-1dab686a9f29\") "
Nov 01 09:45:15.980408 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.979137    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume" pod "cbb7ed3b-4486-44b8-8842-8c857457a124" (UID: "cbb7ed3b-4486-44b8-8842-8c857457a124"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:15.981245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.980820    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume" pod "8c4063bd-75d1-44e0-8885-1dab686a9f29" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:15.981245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.981024    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials" pod "cbb7ed3b-4486-44b8-8842-8c857457a124" (UID: "cbb7ed3b-4486-44b8-8842-8c857457a124"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:15.981990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.981962    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume" pod "cbb7ed3b-4486-44b8-8842-8c857457a124" (UID: "cbb7ed3b-4486-44b8-8842-8c857457a124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:15.982233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.982195    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume" pod "8c4063bd-75d1-44e0-8885-1dab686a9f29" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:15.982381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.982280    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s" pod "cbb7ed3b-4486-44b8-8842-8c857457a124" (UID: "cbb7ed3b-4486-44b8-8842-8c857457a124"). InnerVolumeSpecName "kube-api-access-ggq7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:15.982454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.982432    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:15.982522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.982454    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb7ed3b-4486-44b8-8842-8c857457a124-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:15.982522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.982470    4314 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-routing-overlay-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:15.982522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.982485    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:15.983901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.983880    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb" pod "8c4063bd-75d1-44e0-8885-1dab686a9f29" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29"). InnerVolumeSpecName "kube-api-access-nc5gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:15.984197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:15.984172    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials" pod "8c4063bd-75d1-44e0-8885-1dab686a9f29" (UID: "8c4063bd-75d1-44e0-8885-1dab686a9f29"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:16.084129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.084096    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc5gb\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-kube-api-access-nc5gb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:16.084129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.084123    4314 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c4063bd-75d1-44e0-8885-1dab686a9f29-provider-credentials\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:16.084129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.084133    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggq7s\" (UniqueName: \"kubernetes.io/projected/cbb7ed3b-4486-44b8-8842-8c857457a124-kube-api-access-ggq7s\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:16.084369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.084142    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4063bd-75d1-44e0-8885-1dab686a9f29-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:16.657920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.657880    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerStarted","Data":"32589a24bc9223c83dc076c8685c7124aa065d047ce76a546d971cdcb40aa034"}
Nov 01 09:45:16.661377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.661345    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerStarted","Data":"e340d391b18f507e34449ac37193250f1adb27c084c01f057715a800ec49388c"}
Nov 01 09:45:16.663596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.663573    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerStarted","Data":"d66cd1eed7fb4f075a4f1f038de76835f2d948609b3e17010e61bc4ad5cdee8d"}
Nov 01 09:45:16.665078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.665051    4314 generic.go:356] "Generic (PLEG): container finished" podID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerID="606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7" exitCode=0
Nov 01 09:45:16.665176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.665090    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"
Nov 01 09:45:16.665176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.665125    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" event={"ID":"cbb7ed3b-4486-44b8-8842-8c857457a124","Type":"ContainerDied","Data":"606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7"}
Nov 01 09:45:16.665176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.665163    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk" event={"ID":"cbb7ed3b-4486-44b8-8842-8c857457a124","Type":"ContainerDied","Data":"f53c5252e8991d24dc1541fd3dc4267c06c61473d6d8bbbc4d6d9a0cb9a336db"}
Nov 01 09:45:16.665326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.665178    4314 scope.go:122] "RemoveContainer" containerID="606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7"
Nov 01 09:45:16.667772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.667424    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerStarted","Data":"1dbd22eaff450ef87a2a5a489ebd547772d5e4278fe0a80c73a898cffae6eb81"}
Nov 01 09:45:16.669896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.669873    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerStarted","Data":"fd2790238b1fa0c3d58eeb8250ecd95c6391fd14645bcdf7261d610465939779"}
Nov 01 09:45:16.671526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.671506    4314 generic.go:356] "Generic (PLEG): container finished" podID="8c4063bd-75d1-44e0-8885-1dab686a9f29" containerID="03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e" exitCode=0
Nov 01 09:45:16.671636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.671562    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr" event={"ID":"8c4063bd-75d1-44e0-8885-1dab686a9f29","Type":"ContainerDied","Data":"03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e"}
Nov 01 09:45:16.671636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.671587    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"
Nov 01 09:45:16.671636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.671610    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr" event={"ID":"8c4063bd-75d1-44e0-8885-1dab686a9f29","Type":"ContainerDied","Data":"45af8261a5d1952d938563722f4a108b2d832c3bd952750eadd591c102a71ee4"}
Nov 01 09:45:16.673754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.673733    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerStarted","Data":"0cf1ccafe236729a6e9e76c7486f6bc4415bb6e49b455cc9cec4810bb9c60c7e"}
Nov 01 09:45:16.673946 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.673929    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:16.675086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.675069    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:45:16.677934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.677853    4314 scope.go:122] "RemoveContainer" containerID="606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7"
Nov 01 09:45:16.678175 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:16.678139    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7\": container with ID starting with 606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7 not found: ID does not exist" containerID="606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7"
Nov 01 09:45:16.678232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.678167    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7"} err="failed to get container status \"606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7\": rpc error: code = NotFound desc = could not find container \"606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7\": container with ID starting with 606142ed56b577ab0a8017956fa1c877b13f24ccf243879db802caf6d4c279c7 not found: ID does not exist"
Nov 01 09:45:16.678232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.678184    4314 scope.go:122] "RemoveContainer" containerID="03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e"
Nov 01 09:45:16.688561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.688522    4314 scope.go:122] "RemoveContainer" containerID="03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e"
Nov 01 09:45:16.688835 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:16.688810    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e\": container with ID starting with 03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e not found: ID does not exist" containerID="03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e"
Nov 01 09:45:16.688937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.688841    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e"} err="failed to get container status \"03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e\": rpc error: code = NotFound desc = could not find container \"03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e\": container with ID starting with 03e77135f8954609bb4f76eac97a9c994c6b41e9855310fed6fd8b1f23278a0e not found: ID does not exist"
Nov 01 09:45:16.873955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.873924    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"]
Nov 01 09:45:16.877781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.877755    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d4446d68-dmgvk"]
Nov 01 09:45:16.921434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.921401    4314 scope.go:122] "RemoveContainer" containerID="75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7"
Nov 01 09:45:16.921670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.921647    4314 scope.go:122] "RemoveContainer" containerID="8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05"
Nov 01 09:45:16.931221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:16.931196    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" path="/var/lib/kubelet/pods/cbb7ed3b-4486-44b8-8842-8c857457a124/volumes"
Nov 01 09:45:17.517735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.517703    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"]
Nov 01 09:45:17.521713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.521690    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-fd5d559f7-gmknr"]
Nov 01 09:45:17.589386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.589363    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"]
Nov 01 09:45:17.644448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.644409    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:17.688790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.688753    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerStarted","Data":"eb27b55c28be838bff04e1e93a9c86a4431e7408839228096a31407b4fb3f225"}
Nov 01 09:45:17.691006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.690968    4314 generic.go:356] "Generic (PLEG): container finished" podID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerID="d66cd1eed7fb4f075a4f1f038de76835f2d948609b3e17010e61bc4ad5cdee8d" exitCode=60
Nov 01 09:45:17.691163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.691046    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerDied","Data":"d66cd1eed7fb4f075a4f1f038de76835f2d948609b3e17010e61bc4ad5cdee8d"}
Nov 01 09:45:17.691163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.691116    4314 scope.go:122] "RemoveContainer" containerID="2495961a5ab84725324c748ce76c56b3385f2a021b08c03195314ec11088281d"
Nov 01 09:45:17.691668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.691648    4314 scope.go:122] "RemoveContainer" containerID="d66cd1eed7fb4f075a4f1f038de76835f2d948609b3e17010e61bc4ad5cdee8d"
Nov 01 09:45:17.692058 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:17.692033    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x_odh-ai-gateway-infra(fefe2e59-a43a-4334-8ed2-6f5feca2c3b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0"
Nov 01 09:45:17.693817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.693796    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerStarted","Data":"b27e35e9bae1c572929f89626ce96c24c9be361fd18e9e39c92b8b7ba235eb87"}
Nov 01 09:45:17.697259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.697238    4314 generic.go:356] "Generic (PLEG): container finished" podID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerID="1dbd22eaff450ef87a2a5a489ebd547772d5e4278fe0a80c73a898cffae6eb81" exitCode=60
Nov 01 09:45:17.697373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.697313    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerDied","Data":"1dbd22eaff450ef87a2a5a489ebd547772d5e4278fe0a80c73a898cffae6eb81"}
Nov 01 09:45:17.697667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.697648    4314 scope.go:122] "RemoveContainer" containerID="1dbd22eaff450ef87a2a5a489ebd547772d5e4278fe0a80c73a898cffae6eb81"
Nov 01 09:45:17.697959 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:17.697923    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr_odh-ai-gateway-infra(3f53d4a4-30eb-46f7-b138-ce62135a0086)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086"
Nov 01 09:45:17.699417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.699395    4314 generic.go:356] "Generic (PLEG): container finished" podID="45484203-f964-4732-a284-f501585412ac" containerID="fd2790238b1fa0c3d58eeb8250ecd95c6391fd14645bcdf7261d610465939779" exitCode=60
Nov 01 09:45:17.699508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.699416    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerDied","Data":"fd2790238b1fa0c3d58eeb8250ecd95c6391fd14645bcdf7261d610465939779"}
Nov 01 09:45:17.699762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.699740    4314 scope.go:122] "RemoveContainer" containerID="fd2790238b1fa0c3d58eeb8250ecd95c6391fd14645bcdf7261d610465939779"
Nov 01 09:45:17.700025 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:17.700001    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm_odh-ai-gateway-infra(45484203-f964-4732-a284-f501585412ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" podUID="45484203-f964-4732-a284-f501585412ac"
Nov 01 09:45:17.702725 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.702705    4314 generic.go:356] "Generic (PLEG): container finished" podID="100116a4-7b5a-4083-8d89-ab55162f065b" containerID="0cf1ccafe236729a6e9e76c7486f6bc4415bb6e49b455cc9cec4810bb9c60c7e" exitCode=60
Nov 01 09:45:17.702814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.702770    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerDied","Data":"0cf1ccafe236729a6e9e76c7486f6bc4415bb6e49b455cc9cec4810bb9c60c7e"}
Nov 01 09:45:17.703113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.703082    4314 scope.go:122] "RemoveContainer" containerID="0cf1ccafe236729a6e9e76c7486f6bc4415bb6e49b455cc9cec4810bb9c60c7e"
Nov 01 09:45:17.703340 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:17.703318    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7_odh-ai-gateway-infra(100116a4-7b5a-4083-8d89-ab55162f065b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" podUID="100116a4-7b5a-4083-8d89-ab55162f065b"
Nov 01 09:45:17.704915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.704896    4314 generic.go:356] "Generic (PLEG): container finished" podID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerID="32589a24bc9223c83dc076c8685c7124aa065d047ce76a546d971cdcb40aa034" exitCode=60
Nov 01 09:45:17.705014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.704973    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerDied","Data":"32589a24bc9223c83dc076c8685c7124aa065d047ce76a546d971cdcb40aa034"}
Nov 01 09:45:17.705288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.705270    4314 scope.go:122] "RemoveContainer" containerID="32589a24bc9223c83dc076c8685c7124aa065d047ce76a546d971cdcb40aa034"
Nov 01 09:45:17.705509 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:17.705492    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7_odh-ai-gateway-infra(e73e0470-1f3a-4f32-999a-8527566eea4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e"
Nov 01 09:45:17.707723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.707704    4314 scope.go:122] "RemoveContainer" containerID="330f3a779d075e4a203c6de548c7171340e8dd9a3f47596878c2c071115ca9eb"
Nov 01 09:45:17.707828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.707717    4314 generic.go:356] "Generic (PLEG): container finished" podID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerID="e340d391b18f507e34449ac37193250f1adb27c084c01f057715a800ec49388c" exitCode=60
Nov 01 09:45:17.707828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.707783    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerDied","Data":"e340d391b18f507e34449ac37193250f1adb27c084c01f057715a800ec49388c"}
Nov 01 09:45:17.708381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.708365    4314 scope.go:122] "RemoveContainer" containerID="e340d391b18f507e34449ac37193250f1adb27c084c01f057715a800ec49388c"
Nov 01 09:45:17.708656 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:17.708635    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892105-dlx9j_odh-ai-gateway-infra(e7f9c43d-f941-45f7-b20f-635cd0a83cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5"
Nov 01 09:45:17.721769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.721749    4314 scope.go:122] "RemoveContainer" containerID="97ac9c4475a7b0c71c9b1d867171822fc60126d6345298e39e09543baf08ecc4"
Nov 01 09:45:17.735570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.735524    4314 scope.go:122] "RemoveContainer" containerID="b7bb836ab8e398d3714ace18fe38bf73e12e2afae92d6c205891557651e28b22"
Nov 01 09:45:17.749788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.749771    4314 scope.go:122] "RemoveContainer" containerID="dcf444637a9aa9d56b9ae6f979183da719edcd8c8e4ad9e2ad43e0a66c497c87"
Nov 01 09:45:17.762680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.762665    4314 scope.go:122] "RemoveContainer" containerID="265c6297adaab32a8740dba79cf90e0c5cf9df16adbb5149fa1a1a086132df2c"
Nov 01 09:45:17.779728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.779707    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"]
Nov 01 09:45:17.922256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.922232    4314 scope.go:122] "RemoveContainer" containerID="c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005"
Nov 01 09:45:17.922626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.922343    4314 scope.go:122] "RemoveContainer" containerID="127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec"
Nov 01 09:45:17.925513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.925428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:17.995744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:17.995715    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"]
Nov 01 09:45:18.066642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.066614    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:18.107704 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.107657    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"]
Nov 01 09:45:18.212173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.212141    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"]
Nov 01 09:45:18.224612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.224579    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:18.317677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.317644    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"]
Nov 01 09:45:18.476372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.476340    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"]
Nov 01 09:45:18.493510 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.493480    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:18.718139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.718107    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerStarted","Data":"a71a1a72d08387c9482e8c2c9192885bb32a165d41d0204ff0d80e41a1b90f9c"}
Nov 01 09:45:18.720589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.720526    4314 generic.go:356] "Generic (PLEG): container finished" podID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerID="eb27b55c28be838bff04e1e93a9c86a4431e7408839228096a31407b4fb3f225" exitCode=60
Nov 01 09:45:18.720745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.720584    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerDied","Data":"eb27b55c28be838bff04e1e93a9c86a4431e7408839228096a31407b4fb3f225"}
Nov 01 09:45:18.720745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.720654    4314 scope.go:122] "RemoveContainer" containerID="8147eb6fc0085310aa488862d549941221ee0b2ea209be20a1167fe23f6b7d05"
Nov 01 09:45:18.721162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.721070    4314 scope.go:122] "RemoveContainer" containerID="eb27b55c28be838bff04e1e93a9c86a4431e7408839228096a31407b4fb3f225"
Nov 01 09:45:18.721392 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:18.721352    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz_odh-ai-gateway-infra(c7c4b756-4c9f-4cad-b807-063d3b9fe3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec"
Nov 01 09:45:18.725630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.725601    4314 generic.go:356] "Generic (PLEG): container finished" podID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerID="b27e35e9bae1c572929f89626ce96c24c9be361fd18e9e39c92b8b7ba235eb87" exitCode=60
Nov 01 09:45:18.725779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.725755    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerDied","Data":"b27e35e9bae1c572929f89626ce96c24c9be361fd18e9e39c92b8b7ba235eb87"}
Nov 01 09:45:18.726245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.726217    4314 scope.go:122] "RemoveContainer" containerID="b27e35e9bae1c572929f89626ce96c24c9be361fd18e9e39c92b8b7ba235eb87"
Nov 01 09:45:18.726554 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:18.726512    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn_odh-ai-gateway-infra(34c214a7-4c48-4bda-8fee-d44f737e394c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c"
Nov 01 09:45:18.750594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.745122    4314 scope.go:122] "RemoveContainer" containerID="75e71ca24b457d4718658f8a4775b7738ce5f152a402360e30f33a1ee01db1e7"
Nov 01 09:45:18.755647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.754747    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerStarted","Data":"d498fe6e1347f62b9d513fb52b7d65d5f1bf8ceebf3481df2351ceb988b4886b"}
Nov 01 09:45:18.924030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.924000    4314 scope.go:122] "RemoveContainer" containerID="1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2"
Nov 01 09:45:18.925148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.924762    4314 scope.go:122] "RemoveContainer" containerID="7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33"
Nov 01 09:45:18.925148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.924926    4314 scope.go:122] "RemoveContainer" containerID="57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da"
Nov 01 09:45:18.926029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.925567    4314 scope.go:122] "RemoveContainer" containerID="fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8"
Nov 01 09:45:18.947658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:18.947636    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4063bd-75d1-44e0-8885-1dab686a9f29" path="/var/lib/kubelet/pods/8c4063bd-75d1-44e0-8885-1dab686a9f29/volumes"
Nov 01 09:45:19.228893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.227716    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:19.254201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.254176    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:19.269253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.269224    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:19.282709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.282681    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:19.286391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.285380    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:19.289076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.288955    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337133    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca\") pod \"3f53d4a4-30eb-46f7-b138-ce62135a0086\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337202    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz\" (UniqueName: \"kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz\") pod \"e73e0470-1f3a-4f32-999a-8527566eea4e\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337295    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca\") pod \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337407    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca\") pod \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337442    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m\" (UniqueName: \"kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m\") pod \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\" (UID: \"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337484    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca\") pod \"e73e0470-1f3a-4f32-999a-8527566eea4e\" (UID: \"e73e0470-1f3a-4f32-999a-8527566eea4e\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337505    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k\" (UniqueName: \"kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k\") pod \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\" (UID: \"e7f9c43d-f941-45f7-b20f-635cd0a83cf5\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337653    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8\" (UniqueName: \"kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8\") pod \"45484203-f964-4732-a284-f501585412ac\" (UID: \"45484203-f964-4732-a284-f501585412ac\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337726    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca\") pod \"100116a4-7b5a-4083-8d89-ab55162f065b\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337780    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca\") pod \"45484203-f964-4732-a284-f501585412ac\" (UID: \"45484203-f964-4732-a284-f501585412ac\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337801    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2\" (UniqueName: \"kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2\") pod \"3f53d4a4-30eb-46f7-b138-ce62135a0086\" (UID: \"3f53d4a4-30eb-46f7-b138-ce62135a0086\") "
Nov 01 09:45:19.344554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.337824    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh\" (UniqueName: \"kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh\") pod \"100116a4-7b5a-4083-8d89-ab55162f065b\" (UID: \"100116a4-7b5a-4083-8d89-ab55162f065b\") "
Nov 01 09:45:19.351294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.345840    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca" pod "e73e0470-1f3a-4f32-999a-8527566eea4e" (UID: "e73e0470-1f3a-4f32-999a-8527566eea4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.351294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.347023    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca" pod "3f53d4a4-30eb-46f7-b138-ce62135a0086" (UID: "3f53d4a4-30eb-46f7-b138-ce62135a0086"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.351294 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.348865    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca" pod "e7f9c43d-f941-45f7-b20f-635cd0a83cf5" (UID: "e7f9c43d-f941-45f7-b20f-635cd0a83cf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.367553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.356505    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca" pod "100116a4-7b5a-4083-8d89-ab55162f065b" (UID: "100116a4-7b5a-4083-8d89-ab55162f065b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.367553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.357464    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca" pod "45484203-f964-4732-a284-f501585412ac" (UID: "45484203-f964-4732-a284-f501585412ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.367553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.357698    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca" pod "fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" (UID: "fefe2e59-a43a-4334-8ed2-6f5feca2c3b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.373824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.373783    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8" pod "45484203-f964-4732-a284-f501585412ac" (UID: "45484203-f964-4732-a284-f501585412ac"). InnerVolumeSpecName "kube-api-access-8b4z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.374335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.374299    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh" pod "100116a4-7b5a-4083-8d89-ab55162f065b" (UID: "100116a4-7b5a-4083-8d89-ab55162f065b"). InnerVolumeSpecName "kube-api-access-5p9wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.374473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.374456    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k" pod "e7f9c43d-f941-45f7-b20f-635cd0a83cf5" (UID: "e7f9c43d-f941-45f7-b20f-635cd0a83cf5"). InnerVolumeSpecName "kube-api-access-kwz5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.374989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.374962    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2" pod "3f53d4a4-30eb-46f7-b138-ce62135a0086" (UID: "3f53d4a4-30eb-46f7-b138-ce62135a0086"). InnerVolumeSpecName "kube-api-access-6blt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.375420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.375396    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz" pod "e73e0470-1f3a-4f32-999a-8527566eea4e" (UID: "e73e0470-1f3a-4f32-999a-8527566eea4e"). InnerVolumeSpecName "kube-api-access-t8lxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.378244 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.378215    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m" pod "fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" (UID: "fefe2e59-a43a-4334-8ed2-6f5feca2c3b0"). InnerVolumeSpecName "kube-api-access-4pl2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.439963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.439931    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4z8\" (UniqueName: \"kubernetes.io/projected/45484203-f964-4732-a284-f501585412ac-kube-api-access-8b4z8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.439963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.439959    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/100116a4-7b5a-4083-8d89-ab55162f065b-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.439963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.439969    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45484203-f964-4732-a284-f501585412ac-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.439980    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blt2\" (UniqueName: \"kubernetes.io/projected/3f53d4a4-30eb-46f7-b138-ce62135a0086-kube-api-access-6blt2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.439989    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p9wh\" (UniqueName: \"kubernetes.io/projected/100116a4-7b5a-4083-8d89-ab55162f065b-kube-api-access-5p9wh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.439998    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53d4a4-30eb-46f7-b138-ce62135a0086-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.440006    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lxz\" (UniqueName: \"kubernetes.io/projected/e73e0470-1f3a-4f32-999a-8527566eea4e-kube-api-access-t8lxz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.440015    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.440026    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.440034    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl2m\" (UniqueName: \"kubernetes.io/projected/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0-kube-api-access-4pl2m\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.440044    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73e0470-1f3a-4f32-999a-8527566eea4e-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.440240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.440052    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwz5k\" (UniqueName: \"kubernetes.io/projected/e7f9c43d-f941-45f7-b20f-635cd0a83cf5-kube-api-access-kwz5k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.676767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.674614    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"]
Nov 01 09:45:19.740488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.740448    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:45:19.751267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.751233    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"]
Nov 01 09:45:19.762350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762319    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762367    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762406    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762474    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762547    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762675 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762592    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762680    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762723    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.762867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.762851    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token\") pod \"bf052d21-f5fb-4b68-9f61-705e52111983\" (UID: \"bf052d21-f5fb-4b68-9f61-705e52111983\") "
Nov 01 09:45:19.763172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.763141    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.763302 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.763247    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.763692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.763667    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf052d21-f5fb-4b68-9f61-705e52111983-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.763799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.763696    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.764960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.764811    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.765031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.765009    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.765090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.765059    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.788765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.788734    4314 generic.go:356] "Generic (PLEG): container finished" podID="bf052d21-f5fb-4b68-9f61-705e52111983" containerID="ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c" exitCode=0
Nov 01 09:45:19.788917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.788787    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"
Nov 01 09:45:19.788917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.788831    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" event={"ID":"bf052d21-f5fb-4b68-9f61-705e52111983","Type":"ContainerDied","Data":"ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c"}
Nov 01 09:45:19.788917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.788878    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr" event={"ID":"bf052d21-f5fb-4b68-9f61-705e52111983","Type":"ContainerDied","Data":"bad116392f57d80f1d878da80816c16e5a97ce51cd1d26cabc206f517480cbe7"}
Nov 01 09:45:19.788917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.788906    4314 scope.go:122] "RemoveContainer" containerID="ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c"
Nov 01 09:45:19.799249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.799200    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "kube-api-access-85n59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.799387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.799263    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:45:19.799387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.799315    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.800576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.799787    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x" event={"ID":"fefe2e59-a43a-4334-8ed2-6f5feca2c3b0","Type":"ContainerDied","Data":"a4138ad25158d27217a07159051fe5f3aec0a7c52f56fc3398f29e661ac811c2"}
Nov 01 09:45:19.800576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.799796    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"
Nov 01 09:45:19.802851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.801328    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy" pod "bf052d21-f5fb-4b68-9f61-705e52111983" (UID: "bf052d21-f5fb-4b68-9f61-705e52111983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.806832 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.805737    4314 scope.go:122] "RemoveContainer" containerID="ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c"
Nov 01 09:45:19.809628 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:19.807292    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c\": container with ID starting with ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c not found: ID does not exist" containerID="ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c"
Nov 01 09:45:19.809628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.807347    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c"} err="failed to get container status \"ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c\": rpc error: code = NotFound desc = could not find container \"ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c\": container with ID starting with ad83246a8c5231340914759b02606940dd7886c111f762210f2593fb2d68160c not found: ID does not exist"
Nov 01 09:45:19.809628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.807372    4314 scope.go:122] "RemoveContainer" containerID="d66cd1eed7fb4f075a4f1f038de76835f2d948609b3e17010e61bc4ad5cdee8d"
Nov 01 09:45:19.809628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.809277    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr" event={"ID":"3f53d4a4-30eb-46f7-b138-ce62135a0086","Type":"ContainerDied","Data":"ad46add8f9a238cf47fbbb8c936540f38da8e04fe18b1079eb8ad96aa010dbe1"}
Nov 01 09:45:19.809628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.809402    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"
Nov 01 09:45:19.822505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.822474    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm" event={"ID":"45484203-f964-4732-a284-f501585412ac","Type":"ContainerDied","Data":"2c8f2b558813ada3cd338e9c61fd36ac6a2311683ceeb6a4aeb8f3708329bea2"}
Nov 01 09:45:19.822654 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.822597    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"
Nov 01 09:45:19.827560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.827511    4314 scope.go:122] "RemoveContainer" containerID="1dbd22eaff450ef87a2a5a489ebd547772d5e4278fe0a80c73a898cffae6eb81"
Nov 01 09:45:19.830028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.829151    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerStarted","Data":"9831f6f723833dd9f3bdb3e302e9cc49c65f98061cef83570d69d28cc2c1d707"}
Nov 01 09:45:19.831498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.831470    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:45:19.833326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.833304    4314 generic.go:356] "Generic (PLEG): container finished" podID="34878064-c876-4a53-a449-0ba669063be3" containerID="d498fe6e1347f62b9d513fb52b7d65d5f1bf8ceebf3481df2351ceb988b4886b" exitCode=60
Nov 01 09:45:19.834700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.833371    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerDied","Data":"d498fe6e1347f62b9d513fb52b7d65d5f1bf8ceebf3481df2351ceb988b4886b"}
Nov 01 09:45:19.834700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.833748    4314 scope.go:122] "RemoveContainer" containerID="d498fe6e1347f62b9d513fb52b7d65d5f1bf8ceebf3481df2351ceb988b4886b"
Nov 01 09:45:19.834700 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:19.834062    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x_odh-ai-gateway-infra(34878064-c876-4a53-a449-0ba669063be3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" podUID="34878064-c876-4a53-a449-0ba669063be3"
Nov 01 09:45:19.837402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.837373    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7" event={"ID":"100116a4-7b5a-4083-8d89-ab55162f065b","Type":"ContainerDied","Data":"9e29e02825d5ff5f42b4f1fdff19c4921a5014acd263f712bbba7ad4e22c3425"}
Nov 01 09:45:19.837521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.837490    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"
Nov 01 09:45:19.839668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.839332    4314 generic.go:356] "Generic (PLEG): container finished" podID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerID="b7e15eeed0a0e40b87276e76718c3384c1cd917c84282175dfb0b426808c6240" exitCode=0
Nov 01 09:45:19.839668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.839407    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" event={"ID":"a7be1b21-71cd-4628-a60f-af4599c7ef91","Type":"ContainerDied","Data":"b7e15eeed0a0e40b87276e76718c3384c1cd917c84282175dfb0b426808c6240"}
Nov 01 09:45:19.839668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.839495    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"
Nov 01 09:45:19.864881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.863146    4314 scope.go:122] "RemoveContainer" containerID="fd2790238b1fa0c3d58eeb8250ecd95c6391fd14645bcdf7261d610465939779"
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.866739    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.866797    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.866836    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.866869    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.866927    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.866980    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.867051    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.867142    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.867201    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.867352    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket\") pod \"a7be1b21-71cd-4628-a60f-af4599c7ef91\" (UID: \"a7be1b21-71cd-4628-a60f-af4599c7ef91\") "
Nov 01 09:45:19.867423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.867362    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:19.868121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868080    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7be1b21-71cd-4628-a60f-af4599c7ef91-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868103    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868117    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868133    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868148    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf052d21-f5fb-4b68-9f61-705e52111983-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868161    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85n59\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-kube-api-access-85n59\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868176    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868190    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf052d21-f5fb-4b68-9f61-705e52111983-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.868479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868203    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf052d21-f5fb-4b68-9f61-705e52111983-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.869142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868612    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.869142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.868655    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.869361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.869316    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.871873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.871815    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:19.873518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.872987    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "kube-api-access-jwpdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.873518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.873039    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7" event={"ID":"e73e0470-1f3a-4f32-999a-8527566eea4e","Type":"ContainerDied","Data":"7b78a25faeedb088c3a81694ae61ae2ec927afda9981a523523b1e9b673c5537"}
Nov 01 09:45:19.873518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.873111    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"
Nov 01 09:45:19.876115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.876053    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerStarted","Data":"631cf50b888aaef4a89683b4abd78aa375a2e0a5f06583af553567901775efd3"}
Nov 01 09:45:19.888199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.888031    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j" event={"ID":"e7f9c43d-f941-45f7-b20f-635cd0a83cf5","Type":"ContainerDied","Data":"2e26cb3c2aa39c3ca57889a1ecfc62396557f421f5cb4e71171ca7be44f18f65"}
Nov 01 09:45:19.888199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.888150    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"
Nov 01 09:45:19.900586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.895884    4314 generic.go:356] "Generic (PLEG): container finished" podID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerID="a71a1a72d08387c9482e8c2c9192885bb32a165d41d0204ff0d80e41a1b90f9c" exitCode=60
Nov 01 09:45:19.900586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.896043    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerDied","Data":"a71a1a72d08387c9482e8c2c9192885bb32a165d41d0204ff0d80e41a1b90f9c"}
Nov 01 09:45:19.900586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.897132    4314 scope.go:122] "RemoveContainer" containerID="a71a1a72d08387c9482e8c2c9192885bb32a165d41d0204ff0d80e41a1b90f9c"
Nov 01 09:45:19.900586 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:19.897846    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc_odh-ai-gateway-infra(64975577-698a-42ea-b3d4-e27ebbdd46fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb"
Nov 01 09:45:19.900586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.899640    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:45:19.912842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.912794    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token" pod "a7be1b21-71cd-4628-a60f-af4599c7ef91" (UID: "a7be1b21-71cd-4628-a60f-af4599c7ef91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:19.951573 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.951508    4314 scope.go:122] "RemoveContainer" containerID="127d71691ae8bef2b7f4f1d454056dc6f2331fef1f003d96f5d284788c3a49ec"
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970490    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970519    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970551    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970565    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970581    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpdp\" (UniqueName: \"kubernetes.io/projected/a7be1b21-71cd-4628-a60f-af4599c7ef91-kube-api-access-jwpdp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970596    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.971952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.970609    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7be1b21-71cd-4628-a60f-af4599c7ef91-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.972353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.972055    4314 scope.go:122] "RemoveContainer" containerID="0cf1ccafe236729a6e9e76c7486f6bc4415bb6e49b455cc9cec4810bb9c60c7e"
Nov 01 09:45:19.998783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:19.996848    4314 scope.go:122] "RemoveContainer" containerID="b7e15eeed0a0e40b87276e76718c3384c1cd917c84282175dfb0b426808c6240"
Nov 01 09:45:20.014461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.014434    4314 scope.go:122] "RemoveContainer" containerID="32589a24bc9223c83dc076c8685c7124aa065d047ce76a546d971cdcb40aa034"
Nov 01 09:45:20.027463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.027437    4314 scope.go:122] "RemoveContainer" containerID="e340d391b18f507e34449ac37193250f1adb27c084c01f057715a800ec49388c"
Nov 01 09:45:20.034400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.034372    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:20.096346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.095224    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"]
Nov 01 09:45:20.138253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.138229    4314 scope.go:122] "RemoveContainer" containerID="c1fbea586d0b4ef670671bed16be842216659e8df7e125845656a4d0b61fb005"
Nov 01 09:45:20.183167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.180260    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:20.183167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.182821    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:20.262925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.262894    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:45:20.274730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274705    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.274861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274768    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb\" (UniqueName: \"kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb\") pod \"34c214a7-4c48-4bda-8fee-d44f737e394c\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") "
Nov 01 09:45:20.274861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274812    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.274861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274833    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m\" (UniqueName: \"kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m\") pod \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") "
Nov 01 09:45:20.275015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274894    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.275015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274945    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.275015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.274975    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca\") pod \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\" (UID: \"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec\") "
Nov 01 09:45:20.275186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275018    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.275186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275123    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.275186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275179    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca\") pod \"34c214a7-4c48-4bda-8fee-d44f737e394c\" (UID: \"34c214a7-4c48-4bda-8fee-d44f737e394c\") "
Nov 01 09:45:20.275372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275230    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.275372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275265    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.275372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275319    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket\") pod \"f126f577-c593-4d2f-ba5a-9ad5edd45429\" (UID: \"f126f577-c593-4d2f-ba5a-9ad5edd45429\") "
Nov 01 09:45:20.276168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275781    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca" pod "34c214a7-4c48-4bda-8fee-d44f737e394c" (UID: "34c214a7-4c48-4bda-8fee-d44f737e394c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:20.276168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.275853    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.276168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276026    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.276168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276070    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.276168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276142    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca" pod "c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" (UID: "c7c4b756-4c9f-4cad-b807-063d3b9fe3ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:20.276520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276198    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.276520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276248    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:20.276658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276610    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.276720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276668    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.276720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276685    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.276720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276703    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.276720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276718    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c214a7-4c48-4bda-8fee-d44f737e394c-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.277000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276734    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f126f577-c593-4d2f-ba5a-9ad5edd45429-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.277000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.276748    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.277592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.277565    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "kube-api-access-pb4jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.277796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.277772    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m" pod "c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" (UID: "c7c4b756-4c9f-4cad-b807-063d3b9fe3ec"). InnerVolumeSpecName "kube-api-access-pst7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.277927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.277906    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb" pod "34c214a7-4c48-4bda-8fee-d44f737e394c" (UID: "34c214a7-4c48-4bda-8fee-d44f737e394c"). InnerVolumeSpecName "kube-api-access-dglgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.278136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.278121    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:45:20.278270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.278254    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.278365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.278350    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy" pod "f126f577-c593-4d2f-ba5a-9ad5edd45429" (UID: "f126f577-c593-4d2f-ba5a-9ad5edd45429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.378253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.378223    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglgb\" (UniqueName: \"kubernetes.io/projected/34c214a7-4c48-4bda-8fee-d44f737e394c-kube-api-access-dglgb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.378253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.378255    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4jn\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-kube-api-access-pb4jn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.378446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.378268    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pst7m\" (UniqueName: \"kubernetes.io/projected/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec-kube-api-access-pst7m\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.378446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.378283    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.378446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.378296    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.378446 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.378308    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f126f577-c593-4d2f-ba5a-9ad5edd45429-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.437810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.437771    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"]
Nov 01 09:45:20.657124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.655139    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"]
Nov 01 09:45:20.662288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.662259    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-035625-29892105-b9rgr"]
Nov 01 09:45:20.766648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.766618    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"]
Nov 01 09:45:20.773849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.773822    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:45:20.773983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.773894    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892105-dlx9j"]
Nov 01 09:45:20.877401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.877367    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"]
Nov 01 09:45:20.886203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886175    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.886369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886302    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.886464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886382    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.886464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886439    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.886624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886600    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.886676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886657    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.886878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886853    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.887032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886894    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.887032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886920    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy\") pod \"009c6f51-7dd6-42ad-8edd-8e07a5772267\" (UID: \"009c6f51-7dd6-42ad-8edd-8e07a5772267\") "
Nov 01 09:45:20.887032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.886993    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.887032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887009    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.887266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887215    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.887374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887351    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:20.887598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887574    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.887732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887588    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892105-r2zjm"]
Nov 01 09:45:20.887840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887600    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.887840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887775    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.887840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887792    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.887840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.887808    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009c6f51-7dd6-42ad-8edd-8e07a5772267-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.888703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.888679    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:45:20.889434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.889409    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.889773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.889751    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "kube-api-access-cmxsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.889836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.889799    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy" pod "009c6f51-7dd6-42ad-8edd-8e07a5772267" (UID: "009c6f51-7dd6-42ad-8edd-8e07a5772267"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:45:20.906750 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.906724    4314 generic.go:356] "Generic (PLEG): container finished" podID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerID="9831f6f723833dd9f3bdb3e302e9cc49c65f98061cef83570d69d28cc2c1d707" exitCode=60
Nov 01 09:45:20.906855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.906796    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerDied","Data":"9831f6f723833dd9f3bdb3e302e9cc49c65f98061cef83570d69d28cc2c1d707"}
Nov 01 09:45:20.906921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.906863    4314 scope.go:122] "RemoveContainer" containerID="1a5544a0c439701b561a3916ff96ce070920dfdd1288a15b6fdcb5e4d60068b2"
Nov 01 09:45:20.907243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.907223    4314 scope.go:122] "RemoveContainer" containerID="9831f6f723833dd9f3bdb3e302e9cc49c65f98061cef83570d69d28cc2c1d707"
Nov 01 09:45:20.907624 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:20.907599    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx_odh-ai-gateway-infra(eb94c797-f3f1-449b-8a81-06a3640b45c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2"
Nov 01 09:45:20.912643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.912582    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m" event={"ID":"a7be1b21-71cd-4628-a60f-af4599c7ef91","Type":"ContainerDied","Data":"7ed11b52e560e3f3aec938729e7e5c4ff7112af990001d0c306e5d4bf1eb6416"}
Nov 01 09:45:20.915975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.915954    4314 generic.go:356] "Generic (PLEG): container finished" podID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerID="631cf50b888aaef4a89683b4abd78aa375a2e0a5f06583af553567901775efd3" exitCode=60
Nov 01 09:45:20.916065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.916032    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerDied","Data":"631cf50b888aaef4a89683b4abd78aa375a2e0a5f06583af553567901775efd3"}
Nov 01 09:45:20.916470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.916450    4314 scope.go:122] "RemoveContainer" containerID="631cf50b888aaef4a89683b4abd78aa375a2e0a5f06583af553567901775efd3"
Nov 01 09:45:20.916822 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:20.916802    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg_odh-ai-gateway-infra(cfaca18d-82ea-42ec-a1d9-22e21df4f1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab"
Nov 01 09:45:20.918107 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.918090    4314 generic.go:356] "Generic (PLEG): container finished" podID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerID="54340b801b91827a2b804225673a53fac4dacef26da68e250d34fdb6140e6883" exitCode=60
Nov 01 09:45:20.918191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.918159    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" event={"ID":"b73ccde3-7970-49cc-8dd5-72b119cdcc47","Type":"ContainerDied","Data":"54340b801b91827a2b804225673a53fac4dacef26da68e250d34fdb6140e6883"}
Nov 01 09:45:20.918700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.918681    4314 scope.go:122] "RemoveContainer" containerID="54340b801b91827a2b804225673a53fac4dacef26da68e250d34fdb6140e6883"
Nov 01 09:45:20.918974 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:20.918951    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd_odh-ai-gateway-infra(b73ccde3-7970-49cc-8dd5-72b119cdcc47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47"
Nov 01 09:45:20.924862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.923548    4314 scope.go:122] "RemoveContainer" containerID="fd6c44f9369659a9bdae1b63714c535824f3699d2f3a100773314d62973f69f8"
Nov 01 09:45:20.924862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.924710    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"
Nov 01 09:45:20.927186 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.927163    4314 generic.go:356] "Generic (PLEG): container finished" podID="25b34cb4-e295-466b-90b0-728ba394262f" containerID="65d5a2858c39e7bd74dcc57e5d59d8cd0f5fea00c1844b11d7c4a4d3cf9463cb" exitCode=60
Nov 01 09:45:20.932145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.932114    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"
Nov 01 09:45:20.934386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.934362    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" path="/var/lib/kubelet/pods/3f53d4a4-30eb-46f7-b138-ce62135a0086/volumes"
Nov 01 09:45:20.935150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.935128    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45484203-f964-4732-a284-f501585412ac" path="/var/lib/kubelet/pods/45484203-f964-4732-a284-f501585412ac/volumes"
Nov 01 09:45:20.935755 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.935735    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" path="/var/lib/kubelet/pods/e7f9c43d-f941-45f7-b20f-635cd0a83cf5/volumes"
Nov 01 09:45:20.937036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937014    4314 generic.go:356] "Generic (PLEG): container finished" podID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerID="272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066" exitCode=0
Nov 01 09:45:20.937185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937169    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"
Nov 01 09:45:20.937368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937343    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz" event={"ID":"c7c4b756-4c9f-4cad-b807-063d3b9fe3ec","Type":"ContainerDied","Data":"0fca520e9fee50691f1dea0b3cecfa7a719f098c72bcdd6b2db9ed724ea03142"}
Nov 01 09:45:20.937586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937515    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerDied","Data":"65d5a2858c39e7bd74dcc57e5d59d8cd0f5fea00c1844b11d7c4a4d3cf9463cb"}
Nov 01 09:45:20.937708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937690    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn" event={"ID":"34c214a7-4c48-4bda-8fee-d44f737e394c","Type":"ContainerDied","Data":"f5e02e7ef03a4d81bfb037f9827fbd45ff4a757a6bd7cb7e5c8c05c50ac7e87b"}
Nov 01 09:45:20.937839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937823    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" event={"ID":"f126f577-c593-4d2f-ba5a-9ad5edd45429","Type":"ContainerDied","Data":"272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066"}
Nov 01 09:45:20.937965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.937947    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z" event={"ID":"f126f577-c593-4d2f-ba5a-9ad5edd45429","Type":"ContainerDied","Data":"b9920f58105953c5330f17c9193ca6f144aba8f512988a03eef9c9b75c14ccf9"}
Nov 01 09:45:20.938362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.938348    4314 scope.go:122] "RemoveContainer" containerID="65d5a2858c39e7bd74dcc57e5d59d8cd0f5fea00c1844b11d7c4a4d3cf9463cb"
Nov 01 09:45:20.938666 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:20.938649    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77_odh-ai-gateway-infra(25b34cb4-e295-466b-90b0-728ba394262f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" podUID="25b34cb4-e295-466b-90b0-728ba394262f"
Nov 01 09:45:20.941662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.941603    4314 generic.go:356] "Generic (PLEG): container finished" podID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerID="13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e" exitCode=0
Nov 01 09:45:20.942285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.942136    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" event={"ID":"009c6f51-7dd6-42ad-8edd-8e07a5772267","Type":"ContainerDied","Data":"13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e"}
Nov 01 09:45:20.942285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.942183    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj" event={"ID":"009c6f51-7dd6-42ad-8edd-8e07a5772267","Type":"ContainerDied","Data":"2ad4bac6baf02a348909907ea26ab67cf8d5926c42d17a6d0da11ad85c6fc562"}
Nov 01 09:45:20.942285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.942246    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"
Nov 01 09:45:20.959768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.957519    4314 scope.go:122] "RemoveContainer" containerID="7369299a36457c549433390b188aebb3954be322eefa4d79871ae9c0ce5d7b33"
Nov 01 09:45:20.994279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.990357    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.994279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.990385    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.994279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.990402    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxsm\" (UniqueName: \"kubernetes.io/projected/009c6f51-7dd6-42ad-8edd-8e07a5772267-kube-api-access-cmxsm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.994279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.990417    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.994279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:20.990430    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009c6f51-7dd6-42ad-8edd-8e07a5772267-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.002390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.001409    4314 scope.go:122] "RemoveContainer" containerID="eb27b55c28be838bff04e1e93a9c86a4431e7408839228096a31407b4fb3f225"
Nov 01 09:45:21.018203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.017428    4314 scope.go:122] "RemoveContainer" containerID="57beb5f1f6b12ad1ebae399a10490c4cf6740cb4cbf827462634186d06c928da"
Nov 01 09:45:21.032371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.032342    4314 scope.go:122] "RemoveContainer" containerID="b27e35e9bae1c572929f89626ce96c24c9be361fd18e9e39c92b8b7ba235eb87"
Nov 01 09:45:21.044138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.044113    4314 scope.go:122] "RemoveContainer" containerID="272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066"
Nov 01 09:45:21.061201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.061180    4314 scope.go:122] "RemoveContainer" containerID="272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066"
Nov 01 09:45:21.061880 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:21.061703    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066\": container with ID starting with 272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066 not found: ID does not exist" containerID="272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066"
Nov 01 09:45:21.061880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.061765    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066"} err="failed to get container status \"272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066\": rpc error: code = NotFound desc = could not find container \"272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066\": container with ID starting with 272740a15f8927b85e14467a237be5cd8a379fc9968774e6e1d131fc64cfd066 not found: ID does not exist"
Nov 01 09:45:21.061880 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.061791    4314 scope.go:122] "RemoveContainer" containerID="13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e"
Nov 01 09:45:21.100424 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.100394    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"]
Nov 01 09:45:21.103203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.103179    4314 scope.go:122] "RemoveContainer" containerID="13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e"
Nov 01 09:45:21.103516 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:21.103495    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e\": container with ID starting with 13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e not found: ID does not exist" containerID="13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e"
Nov 01 09:45:21.103589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.103525    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e"} err="failed to get container status \"13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e\": rpc error: code = NotFound desc = could not find container \"13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e\": container with ID starting with 13597aa51d268c46041213e103e96760df2287de92104170e8f0bf72ff870d8e not found: ID does not exist"
Nov 01 09:45:21.105028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.105010    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b3d6bb-29892105-zlpg7"]
Nov 01 09:45:21.105831 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.105813    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:21.138305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.138283    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:21.193655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.193620    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca\") pod \"34878064-c876-4a53-a449-0ba669063be3\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") "
Nov 01 09:45:21.193843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.193761    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq\" (UniqueName: \"kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq\") pod \"34878064-c876-4a53-a449-0ba669063be3\" (UID: \"34878064-c876-4a53-a449-0ba669063be3\") "
Nov 01 09:45:21.193843 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.193782    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9\" (UniqueName: \"kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9\") pod \"64975577-698a-42ea-b3d4-e27ebbdd46fb\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") "
Nov 01 09:45:21.193927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.193861    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca\") pod \"64975577-698a-42ea-b3d4-e27ebbdd46fb\" (UID: \"64975577-698a-42ea-b3d4-e27ebbdd46fb\") "
Nov 01 09:45:21.194017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.193995    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca" pod "34878064-c876-4a53-a449-0ba669063be3" (UID: "34878064-c876-4a53-a449-0ba669063be3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:21.194332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.194309    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca" pod "64975577-698a-42ea-b3d4-e27ebbdd46fb" (UID: "64975577-698a-42ea-b3d4-e27ebbdd46fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:21.194423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.194407    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34878064-c876-4a53-a449-0ba669063be3-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.194481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.194424    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64975577-698a-42ea-b3d4-e27ebbdd46fb-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.196037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.196018    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9" pod "64975577-698a-42ea-b3d4-e27ebbdd46fb" (UID: "64975577-698a-42ea-b3d4-e27ebbdd46fb"). InnerVolumeSpecName "kube-api-access-8s9g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:21.196100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.196043    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq" pod "34878064-c876-4a53-a449-0ba669063be3" (UID: "34878064-c876-4a53-a449-0ba669063be3"). InnerVolumeSpecName "kube-api-access-8p6cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:21.206607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.206580    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"]
Nov 01 09:45:21.217973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.217949    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3a53f2-29892105-cbs4x"]
Nov 01 09:45:21.278929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.278781    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:21.295194 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.295166    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p6cq\" (UniqueName: \"kubernetes.io/projected/34878064-c876-4a53-a449-0ba669063be3-kube-api-access-8p6cq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.295194 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.295193    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9g9\" (UniqueName: \"kubernetes.io/projected/64975577-698a-42ea-b3d4-e27ebbdd46fb-kube-api-access-8s9g9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.431605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.431568    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"]
Nov 01 09:45:21.436561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.436520    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fd5a10-29892105-9nmk7"]
Nov 01 09:45:21.441149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.441126    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"]
Nov 01 09:45:21.442764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.442738    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:21.690914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.690836    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"]
Nov 01 09:45:21.691493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691473    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.691563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691498    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.691563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691513    4314 cpu_manager.go:414] "containerMap: removing container" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.691563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691522    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.691563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691548    4314 cpu_manager.go:414] "containerMap: removing container" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.691563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691555    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691577    4314 cpu_manager.go:414] "containerMap: removing container" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691586    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691604    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691612    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691622    4314 cpu_manager.go:414] "containerMap: removing container" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691629    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691651    4314 cpu_manager.go:414] "containerMap: removing container" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691659    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691674    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691681    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691692    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8c4063bd-75d1-44e0-8885-1dab686a9f29" containerName="payload-processing"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691699    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4063bd-75d1-44e0-8885-1dab686a9f29" containerName="payload-processing"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691713    4314 cpu_manager.go:414] "containerMap: removing container" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.691727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691719    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691732    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691739    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691747    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691755    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691767    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691774    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691789    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691796    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691812    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691819    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691836    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691842    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691856    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691862    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691872    4314 cpu_manager.go:414] "containerMap: removing container" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691878    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691893    4314 cpu_manager.go:414] "containerMap: removing container" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691899    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691910    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691918    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691931    4314 cpu_manager.go:414] "containerMap: removing container" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691938    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691950    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691956    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691967    4314 cpu_manager.go:414] "containerMap: removing container" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691974    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691982    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.691989    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692002    4314 cpu_manager.go:414] "containerMap: removing container" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692009    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692022    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692030    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692045    4314 cpu_manager.go:414] "containerMap: removing container" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692052    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692073    4314 cpu_manager.go:414] "containerMap: removing container" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692080    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692093    4314 cpu_manager.go:414] "containerMap: removing container" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692100    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692115    4314 cpu_manager.go:414] "containerMap: removing container" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692121    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692133    4314 cpu_manager.go:414] "containerMap: removing container" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.692151 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692140    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692319    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692333    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692344    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" containerName="istio-proxy"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692353    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692361    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692404    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692412    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692422    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692430    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb7ed3b-4486-44b8-8842-8c857457a124" containerName="payload-processing"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692441    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692449    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692457    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692464    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f53d4a4-30eb-46f7-b138-ce62135a0086" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692472    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692483    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" containerName="istio-proxy"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692494    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692506    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692518    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692527    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692552    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692562    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="45484203-f964-4732-a284-f501585412ac" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692571    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692578    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692588    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" containerName="istio-proxy"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692596    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692606    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692616    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" containerName="istio-proxy"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692626    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4063bd-75d1-44e0-8885-1dab686a9f29" containerName="payload-processing"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692634    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.693398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.692644    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.698034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.698010    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.746742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.746716    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3710de\""
Nov 01 09:45:21.746920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.746904    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3710de-dockercfg-f865f\""
Nov 01 09:45:21.747362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.747342    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"]
Nov 01 09:45:21.802583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.802527    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbg4d\" (UniqueName: \"kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d\") pod \"payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.802769 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.802708    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.891825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.891790    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"]
Nov 01 09:45:21.893055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893009    4314 cpu_manager.go:414] "containerMap: removing container" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.893288 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893254    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" containerName="cleanup"
Nov 01 09:45:21.893466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893454    4314 cpu_manager.go:414] "containerMap: removing container" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.893605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893580    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.893730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893720    4314 cpu_manager.go:414] "containerMap: removing container" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.893870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893859    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.893982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.893972    4314 cpu_manager.go:414] "containerMap: removing container" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.894070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894060    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.894179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894165    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.894277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894266    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" containerName="cleanup"
Nov 01 09:45:21.894543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894519    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" containerName="cleanup"
Nov 01 09:45:21.894642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894632    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" containerName="cleanup"
Nov 01 09:45:21.894722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894714    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34878064-c876-4a53-a449-0ba669063be3" containerName="cleanup"
Nov 01 09:45:21.894803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894795    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" containerName="cleanup"
Nov 01 09:45:21.894895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894885    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" containerName="cleanup"
Nov 01 09:45:21.894968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.894959    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f9c43d-f941-45f7-b20f-635cd0a83cf5" containerName="cleanup"
Nov 01 09:45:21.899591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.899565    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:21.904383 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.904354    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbg4d\" (UniqueName: \"kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d\") pod \"payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.904521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.904504    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.905195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.905173    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.912127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.912103    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"]
Nov 01 09:45:21.961526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.961464    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbg4d\" (UniqueName: \"kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d\") pod \"payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:21.980388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.980361    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"
Nov 01 09:45:21.980522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.980396    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc" event={"ID":"64975577-698a-42ea-b3d4-e27ebbdd46fb","Type":"ContainerDied","Data":"02811a6560d8a533ebb5986f9a3f26ce708ab99e75c70ef31f3ff5576d622dc6"}
Nov 01 09:45:21.980522 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.980463    4314 scope.go:122] "RemoveContainer" containerID="a71a1a72d08387c9482e8c2c9192885bb32a165d41d0204ff0d80e41a1b90f9c"
Nov 01 09:45:21.996577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.996555    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"
Nov 01 09:45:21.996577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.996562    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x" event={"ID":"34878064-c876-4a53-a449-0ba669063be3","Type":"ContainerDied","Data":"ada7efbe47796b266299fdafe8c2b93b26269a6f3efbcae7a0f756c52ca24f1f"}
Nov 01 09:45:21.996734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:21.996621    4314 scope.go:122] "RemoveContainer" containerID="d498fe6e1347f62b9d513fb52b7d65d5f1bf8ceebf3481df2351ceb988b4886b"
Nov 01 09:45:22.005599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.005572    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.005797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.005764    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5nl9\" (UniqueName: \"kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9\") pod \"payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.013596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.013573    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:22.108576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.107079    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5nl9\" (UniqueName: \"kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9\") pod \"payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.108576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.107308    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.108576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.108262    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.154637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.154605    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"]
Nov 01 09:45:22.157828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.157799    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:22.158224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.158203    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-cqw6z"]
Nov 01 09:45:22.170484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.170455    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5nl9\" (UniqueName: \"kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9\") pod \"payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.209991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.208259    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca\") pod \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") "
Nov 01 09:45:22.209991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.209144    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d\" (UniqueName: \"kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d\") pod \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\" (UID: \"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab\") "
Nov 01 09:45:22.209991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.209125    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca" pod "cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" (UID: "cfaca18d-82ea-42ec-a1d9-22e21df4f1ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:22.209991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.209943    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:22.212372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.212316    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d" pod "cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" (UID: "cfaca18d-82ea-42ec-a1d9-22e21df4f1ab"). InnerVolumeSpecName "kube-api-access-zb29d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:22.218479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.218457    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:22.311122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.311093    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb29d\" (UniqueName: \"kubernetes.io/projected/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab-kube-api-access-zb29d\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:22.324577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.324547    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"]
Nov 01 09:45:22.331936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.331910    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vwzjj"]
Nov 01 09:45:22.338243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.338217    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"]
Nov 01 09:45:22.400662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.400629    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"]
Nov 01 09:45:22.401992 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:22.401964    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1638a63_b184_4a57_b409_fdb0ca0b6689.slice/crio-8efc5e11e7042f9b8ec6dfd8f6e291b01b769dbc83376fc97efd8cda8a7f79d6 WatchSource:0}: Error finding container 8efc5e11e7042f9b8ec6dfd8f6e291b01b769dbc83376fc97efd8cda8a7f79d6: Status 404 returned error can't find the container with id 8efc5e11e7042f9b8ec6dfd8f6e291b01b769dbc83376fc97efd8cda8a7f79d6
Nov 01 09:45:22.434180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.434141    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"]
Nov 01 09:45:22.440046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.440015    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qj28m"]
Nov 01 09:45:22.477238 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.477215    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:22.513806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.513782    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca\") pod \"eb94c797-f3f1-449b-8a81-06a3640b45c2\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") "
Nov 01 09:45:22.513920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.513832    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq\" (UniqueName: \"kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq\") pod \"eb94c797-f3f1-449b-8a81-06a3640b45c2\" (UID: \"eb94c797-f3f1-449b-8a81-06a3640b45c2\") "
Nov 01 09:45:22.514139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.514109    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca" pod "eb94c797-f3f1-449b-8a81-06a3640b45c2" (UID: "eb94c797-f3f1-449b-8a81-06a3640b45c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:22.514340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.514322    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb94c797-f3f1-449b-8a81-06a3640b45c2-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:22.515765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.515748    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq" pod "eb94c797-f3f1-449b-8a81-06a3640b45c2" (UID: "eb94c797-f3f1-449b-8a81-06a3640b45c2"). InnerVolumeSpecName "kube-api-access-rdknq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:22.590079 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.590045    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s" podUID="1e0a37f6-2b52-4910-8cf7-0d5971516d82" containerName="payload-processing" containerID="cri-o://1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69" gracePeriod=25
Nov 01 09:45:22.590574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.590545    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"]
Nov 01 09:45:22.591076 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:22.591052    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea85d98_ce91_4bd7_b225_1cc0a8553c72.slice/crio-6361aa1808e235cbcd93767eaa34efaf5ba439f7a9fd7d34688fdf3227a43637 WatchSource:0}: Error finding container 6361aa1808e235cbcd93767eaa34efaf5ba439f7a9fd7d34688fdf3227a43637: Status 404 returned error can't find the container with id 6361aa1808e235cbcd93767eaa34efaf5ba439f7a9fd7d34688fdf3227a43637
Nov 01 09:45:22.615514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.615486    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdknq\" (UniqueName: \"kubernetes.io/projected/eb94c797-f3f1-449b-8a81-06a3640b45c2-kube-api-access-rdknq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:22.818491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.818455    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"]
Nov 01 09:45:22.821932 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.821907    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49f18f-29892105-mszpz"]
Nov 01 09:45:22.872458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.872434    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:45:22.877166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.877145    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"]
Nov 01 09:45:22.918153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.918122    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"]
Nov 01 09:45:22.920123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.920099    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx\" (UniqueName: \"kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx\") pod \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") "
Nov 01 09:45:22.920866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.920834    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume\") pod \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\" (UID: \"1e0a37f6-2b52-4910-8cf7-0d5971516d82\") "
Nov 01 09:45:22.921607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.921564    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume" pod "1e0a37f6-2b52-4910-8cf7-0d5971516d82" (UID: "1e0a37f6-2b52-4910-8cf7-0d5971516d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:22.922582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.921808    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0a37f6-2b52-4910-8cf7-0d5971516d82-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:22.923099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.923072    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx" pod "1e0a37f6-2b52-4910-8cf7-0d5971516d82" (UID: "1e0a37f6-2b52-4910-8cf7-0d5971516d82"). InnerVolumeSpecName "kube-api-access-7mnfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:22.932924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.932887    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009c6f51-7dd6-42ad-8edd-8e07a5772267" path="/var/lib/kubelet/pods/009c6f51-7dd6-42ad-8edd-8e07a5772267/volumes"
Nov 01 09:45:22.933443 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.933426    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100116a4-7b5a-4083-8d89-ab55162f065b" path="/var/lib/kubelet/pods/100116a4-7b5a-4083-8d89-ab55162f065b/volumes"
Nov 01 09:45:22.934090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.934071    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7be1b21-71cd-4628-a60f-af4599c7ef91" path="/var/lib/kubelet/pods/a7be1b21-71cd-4628-a60f-af4599c7ef91/volumes"
Nov 01 09:45:22.935080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.935059    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c4b756-4c9f-4cad-b807-063d3b9fe3ec" path="/var/lib/kubelet/pods/c7c4b756-4c9f-4cad-b807-063d3b9fe3ec/volumes"
Nov 01 09:45:22.938858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.935718    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73e0470-1f3a-4f32-999a-8527566eea4e" path="/var/lib/kubelet/pods/e73e0470-1f3a-4f32-999a-8527566eea4e/volumes"
Nov 01 09:45:22.938858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.936320    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f126f577-c593-4d2f-ba5a-9ad5edd45429" path="/var/lib/kubelet/pods/f126f577-c593-4d2f-ba5a-9ad5edd45429/volumes"
Nov 01 09:45:22.938858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.936906    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefe2e59-a43a-4334-8ed2-6f5feca2c3b0" path="/var/lib/kubelet/pods/fefe2e59-a43a-4334-8ed2-6f5feca2c3b0/volumes"
Nov 01 09:45:22.938858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:22.937517    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-mhmbr"]
Nov 01 09:45:23.008754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.008731    4314 generic.go:356] "Generic (PLEG): container finished" podID="1e0a37f6-2b52-4910-8cf7-0d5971516d82" containerID="1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69" exitCode=0
Nov 01 09:45:23.009144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.008778    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"
Nov 01 09:45:23.009144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.008811    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s" event={"ID":"1e0a37f6-2b52-4910-8cf7-0d5971516d82","Type":"ContainerDied","Data":"1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69"}
Nov 01 09:45:23.009144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.008869    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s" event={"ID":"1e0a37f6-2b52-4910-8cf7-0d5971516d82","Type":"ContainerDied","Data":"aababb2722d0821ec2fc2db91587ed29344fb0a4ba7b715e956b076e0ddc7710"}
Nov 01 09:45:23.009144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.008902    4314 scope.go:122] "RemoveContainer" containerID="1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69"
Nov 01 09:45:23.011749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.011728    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77" event={"ID":"25b34cb4-e295-466b-90b0-728ba394262f","Type":"ContainerDied","Data":"ab7471b802d3aee7dac586e0941377f8dcde21f07904f3ba697bdb0e2c1de8e5"}
Nov 01 09:45:23.011872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.011772    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab7471b802d3aee7dac586e0941377f8dcde21f07904f3ba697bdb0e2c1de8e5"
Nov 01 09:45:23.013515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.013488    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6" event={"ID":"d1638a63-b184-4a57-b409-fdb0ca0b6689","Type":"ContainerStarted","Data":"6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db"}
Nov 01 09:45:23.013619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.013551    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6" event={"ID":"d1638a63-b184-4a57-b409-fdb0ca0b6689","Type":"ContainerStarted","Data":"8efc5e11e7042f9b8ec6dfd8f6e291b01b769dbc83376fc97efd8cda8a7f79d6"}
Nov 01 09:45:23.013619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.013581    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:23.014923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.014904    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"
Nov 01 09:45:23.015027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.014903    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx" event={"ID":"eb94c797-f3f1-449b-8a81-06a3640b45c2","Type":"ContainerDied","Data":"2c841880cf6a4243fe602f0e1c50d364ed773948228fc6b47059d2b3a966d368"}
Nov 01 09:45:23.017623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.017592    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" event={"ID":"0ea85d98-ce91-4bd7-b225-1cc0a8553c72","Type":"ContainerStarted","Data":"02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0"}
Nov 01 09:45:23.017727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.017657    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:23.017727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.017688    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" event={"ID":"0ea85d98-ce91-4bd7-b225-1cc0a8553c72","Type":"ContainerStarted","Data":"6361aa1808e235cbcd93767eaa34efaf5ba439f7a9fd7d34688fdf3227a43637"}
Nov 01 09:45:23.018981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.018958    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg" event={"ID":"cfaca18d-82ea-42ec-a1d9-22e21df4f1ab","Type":"ContainerDied","Data":"a8e42e27e6813ebdf1e80f5aa7c26e246f99a2c280c19a11792b5e23b8327205"}
Nov 01 09:45:23.019080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.019001    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"
Nov 01 09:45:23.022591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.022569    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mnfx\" (UniqueName: \"kubernetes.io/projected/1e0a37f6-2b52-4910-8cf7-0d5971516d82-kube-api-access-7mnfx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:23.023841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.023818    4314 scope.go:122] "RemoveContainer" containerID="1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69"
Nov 01 09:45:23.024205 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:23.024170    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69\": container with ID starting with 1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69 not found: ID does not exist" containerID="1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69"
Nov 01 09:45:23.024310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.024217    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69"} err="failed to get container status \"1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69\": rpc error: code = NotFound desc = could not find container \"1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69\": container with ID starting with 1e7b1a214abf2949a0c0ab0c77e44cd77c3135bee4b535d14d128fe8c891ce69 not found: ID does not exist"
Nov 01 09:45:23.024310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.024237    4314 scope.go:122] "RemoveContainer" containerID="9831f6f723833dd9f3bdb3e302e9cc49c65f98061cef83570d69d28cc2c1d707"
Nov 01 09:45:23.026673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.026654    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:23.034917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.034899    4314 scope.go:122] "RemoveContainer" containerID="631cf50b888aaef4a89683b4abd78aa375a2e0a5f06583af553567901775efd3"
Nov 01 09:45:23.123418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.123389    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw\" (UniqueName: \"kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw\") pod \"25b34cb4-e295-466b-90b0-728ba394262f\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") "
Nov 01 09:45:23.123575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.123561    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca\") pod \"25b34cb4-e295-466b-90b0-728ba394262f\" (UID: \"25b34cb4-e295-466b-90b0-728ba394262f\") "
Nov 01 09:45:23.123935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.123908    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca" pod "25b34cb4-e295-466b-90b0-728ba394262f" (UID: "25b34cb4-e295-466b-90b0-728ba394262f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:23.125765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.125743    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw" pod "25b34cb4-e295-466b-90b0-728ba394262f" (UID: "25b34cb4-e295-466b-90b0-728ba394262f"). InnerVolumeSpecName "kube-api-access-r2zvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:23.193805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.193776    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"]
Nov 01 09:45:23.198243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.198220    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-58de5a-29892105-6p7tn"]
Nov 01 09:45:23.224777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.224754    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zvw\" (UniqueName: \"kubernetes.io/projected/25b34cb4-e295-466b-90b0-728ba394262f-kube-api-access-r2zvw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:23.224777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.224776    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b34cb4-e295-466b-90b0-728ba394262f-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:23.359524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.359444    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"]
Nov 01 09:45:23.366211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.366186    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-12a01a-6766dbb589-r2c2s"]
Nov 01 09:45:23.465656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.465627    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"]
Nov 01 09:45:23.469700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.469678    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892105-7fh2x"]
Nov 01 09:45:23.735864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.735834    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"]
Nov 01 09:45:23.741553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.741517    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-54e01b-29892105-h84tg"]
Nov 01 09:45:23.852331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.852268    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" podStartSLOduration=2.852255001 podStartE2EDuration="2.852255001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:23.849283329 +0000 UTC m=+4397.043677883" watchObservedRunningTime="2026-11-01 09:45:23.852255001 +0000 UTC m=+4397.046649567"
Nov 01 09:45:23.957709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:23.957651    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6" podStartSLOduration=2.957631585 podStartE2EDuration="2.957631585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:23.95557669 +0000 UTC m=+4397.149971245" watchObservedRunningTime="2026-11-01 09:45:23.957631585 +0000 UTC m=+4397.152026143"
Nov 01 09:45:24.030697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.030612    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"
Nov 01 09:45:24.038008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.037982    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"]
Nov 01 09:45:24.063808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.063764    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"]
Nov 01 09:45:24.069421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.069394    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf99-29892105-k65lc"]
Nov 01 09:45:24.225114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.225079    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:24.238586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.238553    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"]
Nov 01 09:45:24.241913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.241887    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cddc5b-29892105-vhzwx"]
Nov 01 09:45:24.337824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.337741    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj\" (UniqueName: \"kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj\") pod \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") "
Nov 01 09:45:24.337988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.337889    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca\") pod \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\" (UID: \"b73ccde3-7970-49cc-8dd5-72b119cdcc47\") "
Nov 01 09:45:24.338259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.338234    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca" pod "b73ccde3-7970-49cc-8dd5-72b119cdcc47" (UID: "b73ccde3-7970-49cc-8dd5-72b119cdcc47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:24.340047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.340027    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj" pod "b73ccde3-7970-49cc-8dd5-72b119cdcc47" (UID: "b73ccde3-7970-49cc-8dd5-72b119cdcc47"). InnerVolumeSpecName "kube-api-access-knppj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:24.439811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.439778    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knppj\" (UniqueName: \"kubernetes.io/projected/b73ccde3-7970-49cc-8dd5-72b119cdcc47-kube-api-access-knppj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:24.439811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.439805    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73ccde3-7970-49cc-8dd5-72b119cdcc47-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:24.449738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.449705    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"]
Nov 01 09:45:24.453412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.453386    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-12a01a-29892105-pvs77"]
Nov 01 09:45:24.478612 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.478584    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:24.932560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.932492    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0a37f6-2b52-4910-8cf7-0d5971516d82" path="/var/lib/kubelet/pods/1e0a37f6-2b52-4910-8cf7-0d5971516d82/volumes"
Nov 01 09:45:24.932949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.932934    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b34cb4-e295-466b-90b0-728ba394262f" path="/var/lib/kubelet/pods/25b34cb4-e295-466b-90b0-728ba394262f/volumes"
Nov 01 09:45:24.933313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.933301    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34878064-c876-4a53-a449-0ba669063be3" path="/var/lib/kubelet/pods/34878064-c876-4a53-a449-0ba669063be3/volumes"
Nov 01 09:45:24.933996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.933981    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c214a7-4c48-4bda-8fee-d44f737e394c" path="/var/lib/kubelet/pods/34c214a7-4c48-4bda-8fee-d44f737e394c/volumes"
Nov 01 09:45:24.934356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.934344    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64975577-698a-42ea-b3d4-e27ebbdd46fb" path="/var/lib/kubelet/pods/64975577-698a-42ea-b3d4-e27ebbdd46fb/volumes"
Nov 01 09:45:24.934792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.934780    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf052d21-f5fb-4b68-9f61-705e52111983" path="/var/lib/kubelet/pods/bf052d21-f5fb-4b68-9f61-705e52111983/volumes"
Nov 01 09:45:24.935422 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.935407    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" path="/var/lib/kubelet/pods/cfaca18d-82ea-42ec-a1d9-22e21df4f1ab/volumes"
Nov 01 09:45:24.935788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:24.935776    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" path="/var/lib/kubelet/pods/eb94c797-f3f1-449b-8a81-06a3640b45c2/volumes"
Nov 01 09:45:25.039471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.039440    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"
Nov 01 09:45:25.039924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.039466    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd" event={"ID":"b73ccde3-7970-49cc-8dd5-72b119cdcc47","Type":"ContainerDied","Data":"5a8b0750883897dfb8e240c4214fa016dfc091d4980d2df07006bf8d3ca263eb"}
Nov 01 09:45:25.039924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.039564    4314 scope.go:122] "RemoveContainer" containerID="54340b801b91827a2b804225673a53fac4dacef26da68e250d34fdb6140e6883"
Nov 01 09:45:25.084248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.084215    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:25.197072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.196998    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"]
Nov 01 09:45:25.242773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.242742    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c81a9d-29892105-t92zd"]
Nov 01 09:45:25.391764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:25.390471    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:26.932105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:26.932071    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" path="/var/lib/kubelet/pods/b73ccde3-7970-49cc-8dd5-72b119cdcc47/volumes"
Nov 01 09:45:29.401788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.401754    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"]
Nov 01 09:45:29.402385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402369    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.402428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402387    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.402428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402401    4314 cpu_manager.go:414] "containerMap: removing container" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.402428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402406    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.402428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402418    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.402428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402424    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402431    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402436    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402448    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402453    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402462    4314 cpu_manager.go:414] "containerMap: removing container" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402467    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402473    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402478    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402487    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402491    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402499    4314 cpu_manager.go:414] "containerMap: removing container" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402503    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402523    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1e0a37f6-2b52-4910-8cf7-0d5971516d82" containerName="payload-processing"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402541    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0a37f6-2b52-4910-8cf7-0d5971516d82" containerName="payload-processing"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402551    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.402638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402555    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402650    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402658    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402667    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0a37f6-2b52-4910-8cf7-0d5971516d82" containerName="payload-processing"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402673    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402678    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402683    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402690    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402697    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402703    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402709    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.403099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.402715    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.407753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.407735    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.448266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.448229    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"]
Nov 01 09:45:29.449131 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.449111    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.449178 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.449135    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaca18d-82ea-42ec-a1d9-22e21df4f1ab" containerName="cleanup"
Nov 01 09:45:29.449214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.449179    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.449214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.449188    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73ccde3-7970-49cc-8dd5-72b119cdcc47" containerName="cleanup"
Nov 01 09:45:29.449378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.449365    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb94c797-f3f1-449b-8a81-06a3640b45c2" containerName="cleanup"
Nov 01 09:45:29.449429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.449382    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b34cb4-e295-466b-90b0-728ba394262f" containerName="cleanup"
Nov 01 09:45:29.453450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.453434    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"]
Nov 01 09:45:29.453592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.453573    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.457557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.457524    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.464080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.464058    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"]
Nov 01 09:45:29.468004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.467985    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.490604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.490573    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\""
Nov 01 09:45:29.501393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.501359    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.501561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.501466    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.501561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.501542    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.501720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.501620    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5tr\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.501783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.501766    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.501895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.501869    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.502043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.502020    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.502117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.502071    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.502170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.502113    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.503199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.503179    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"]
Nov 01 09:45:29.550859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.550789    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"]
Nov 01 09:45:29.557367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.557333    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"]
Nov 01 09:45:29.568101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.568066    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"]
Nov 01 09:45:29.602959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.602927    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.603146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.602976    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.603146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.603146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603029    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.603146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603051    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxj7\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.603373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603150    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6xj\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.603373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603233    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.603373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603281    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.603373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603301    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.603373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603326    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.603373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603354    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.603630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603414    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.603630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603439    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.603630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603478    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.603630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603564    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.603630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603606    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.603851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603634    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.603851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603761    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.603851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603762    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.603851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603819    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.603983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603891    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.604016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.603977    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.604053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604027    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604038    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604117 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604053    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.604153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604112    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.604153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604135    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.604257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604164    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.604321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604259    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604319    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.604390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604360    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.604499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604389    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.604499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604436    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604463    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.604499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604486    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5tr\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604506    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.604726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604562    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.604726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604603    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604650    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.604726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604703    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9nn\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.605003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.604928    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.606595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.606568    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.606734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.606606    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.660485 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.660422    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.660912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.660888    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5tr\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4x6zz\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.706460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706424    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.706460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706461    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.706716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706489    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.706716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706587    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.706716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706621    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.706716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706649    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.706716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706682    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706719    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706743    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706810    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706836    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706869    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706900    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706946    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706962    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.706995    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707023    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707054    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707092    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707122    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707173    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9nn\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707243    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707251    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707328    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707356    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707386    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxj7\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707421    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6xj\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707430    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707438    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707459    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.707570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707554    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707581    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707756    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707844    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.707911    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.708137    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.708174    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.708259    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.708420    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.708629    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.709082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.708908    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.709659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.709479    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.710229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.710203    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.710333 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.710249    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.710520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.710499    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.710828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.710808    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.710990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.710970    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.721993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.721974    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:29.765052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.765015    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.765473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.765447    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.765594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.765569    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.765702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.765680    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxj7\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7\") pod \"maas-default-gateway-openshift-default-746ff8f49f-765gj\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.765758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.765738    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9nn\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-tt7l8\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.766124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.766104    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6xj\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bn6q8\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:29.769607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.769555    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:29.776106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.776084    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:29.789925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:29.789583    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:30.142428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.142394    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"]
Nov 01 09:45:30.142855 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:30.142826    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8837ef1f_f84f_484d_89b4_49ef01386c8c.slice/crio-d695316720084ee2db956091ffba4a7d7500677bd6b6676510e85a99d35f64c0 WatchSource:0}: Error finding container d695316720084ee2db956091ffba4a7d7500677bd6b6676510e85a99d35f64c0: Status 404 returned error can't find the container with id d695316720084ee2db956091ffba4a7d7500677bd6b6676510e85a99d35f64c0
Nov 01 09:45:30.145507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.145473    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.145611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.145592    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.145662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.145627    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.192081 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:30.192054    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c48ed1_1560_4392_ab4d_11622052f0f1.slice/crio-cd06c45d5ebc9b8f6677e8c4bcdcde031966eabf9d7c87cf68af7b35cc0f6f14 WatchSource:0}: Error finding container cd06c45d5ebc9b8f6677e8c4bcdcde031966eabf9d7c87cf68af7b35cc0f6f14: Status 404 returned error can't find the container with id cd06c45d5ebc9b8f6677e8c4bcdcde031966eabf9d7c87cf68af7b35cc0f6f14
Nov 01 09:45:30.192193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.192170    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"]
Nov 01 09:45:30.195110 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.195070    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.195190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.195163    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.195240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.195214    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.241137 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:30.240703    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f19af84_35e1_4ea1_97d9_68396e7099f1.slice/crio-35fb55ce6a4c5ede93a56bfad6da6ad9d9daa35b46ecead0fff925da743b2fb1 WatchSource:0}: Error finding container 35fb55ce6a4c5ede93a56bfad6da6ad9d9daa35b46ecead0fff925da743b2fb1: Status 404 returned error can't find the container with id 35fb55ce6a4c5ede93a56bfad6da6ad9d9daa35b46ecead0fff925da743b2fb1
Nov 01 09:45:30.242240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.242214    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"]
Nov 01 09:45:30.244670 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.244625    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.244798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.244722    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.244798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.244767    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.246894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.246870    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"]
Nov 01 09:45:30.247024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.246995    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.247159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.247070    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.247159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.247118    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:45:30.816656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:30.816618    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:31.089377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.089291    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" event={"ID":"7f19af84-35e1-4ea1-97d9-68396e7099f1","Type":"ContainerStarted","Data":"71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4"}
Nov 01 09:45:31.089377 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.089351    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" event={"ID":"7f19af84-35e1-4ea1-97d9-68396e7099f1","Type":"ContainerStarted","Data":"35fb55ce6a4c5ede93a56bfad6da6ad9d9daa35b46ecead0fff925da743b2fb1"}
Nov 01 09:45:31.091106 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.091073    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" event={"ID":"8837ef1f-f84f-484d-89b4-49ef01386c8c","Type":"ContainerStarted","Data":"c08af440eaa276c7f542d9b0825c06b6fcef4d1e31528122d9396dfe53471de2"}
Nov 01 09:45:31.091246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.091123    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" event={"ID":"8837ef1f-f84f-484d-89b4-49ef01386c8c","Type":"ContainerStarted","Data":"d695316720084ee2db956091ffba4a7d7500677bd6b6676510e85a99d35f64c0"}
Nov 01 09:45:31.092733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.092705    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" event={"ID":"57c48ed1-1560-4392-ab4d-11622052f0f1","Type":"ContainerStarted","Data":"6803bb5a13ae3d628068d5f1d5f7b2d3f95e322d3a32c64767eda0c36a46cc74"}
Nov 01 09:45:31.092845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.092748    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" event={"ID":"57c48ed1-1560-4392-ab4d-11622052f0f1","Type":"ContainerStarted","Data":"cd06c45d5ebc9b8f6677e8c4bcdcde031966eabf9d7c87cf68af7b35cc0f6f14"}
Nov 01 09:45:31.094169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.094147    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" event={"ID":"32b83241-0e9e-4599-83c6-77ac7521427e","Type":"ContainerStarted","Data":"d58b913daad100c8b6f27371319e22dbaeaf106c375c8ffbe67f493bb79d8f5d"}
Nov 01 09:45:31.094266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.094183    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" event={"ID":"32b83241-0e9e-4599-83c6-77ac7521427e","Type":"ContainerStarted","Data":"8500a300063d95b6fb83d1da8683a25bb499cb4042eb18531fc5de0160159c7a"}
Nov 01 09:45:31.207274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.207191    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podStartSLOduration=2.207165901 podStartE2EDuration="2.207165901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:31.201824589 +0000 UTC m=+4404.396219149" watchObservedRunningTime="2026-11-01 09:45:31.207165901 +0000 UTC m=+4404.401560461"
Nov 01 09:45:31.332402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.330086    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podStartSLOduration=2.3300642639999998 podStartE2EDuration="2.330064264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:31.325601513 +0000 UTC m=+4404.519996067" watchObservedRunningTime="2026-11-01 09:45:31.330064264 +0000 UTC m=+4404.524458818"
Nov 01 09:45:31.500619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.452504    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podStartSLOduration=2.452483735 podStartE2EDuration="2.452483735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:31.450036215 +0000 UTC m=+4404.644430785" watchObservedRunningTime="2026-11-01 09:45:31.452483735 +0000 UTC m=+4404.646878291"
Nov 01 09:45:31.640583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.602480    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podStartSLOduration=2.602460017 podStartE2EDuration="2.602460017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:31.599546628 +0000 UTC m=+4404.793941175" watchObservedRunningTime="2026-11-01 09:45:31.602460017 +0000 UTC m=+4404.796854575"
Nov 01 09:45:31.737581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.737547    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:31.754762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.754664    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:31.755012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.754985    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:31.771016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.770982    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:31.780705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.780664    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:31.780970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.780937    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:31.781509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.781486    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:31.795705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.795667    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:31.795983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.795956    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:31.796628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.796601    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:31.819114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.819078    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:31.819741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:31.819704    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:32.728245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.728206    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:32.728631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.728597    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:32.773041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.773000    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:32.773328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.773297    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:32.777913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.777884    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:32.778111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.778087    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:32.793945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.793911    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:32.794162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.794137    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:32.846705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.846616    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 09:45:32.848606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.848576    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 09:45:32.880569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.876662    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:32.893571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:32.891662    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:33.277592 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.268578    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 09:45:33.309559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.308616    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:33.309772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.309664    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:33.316637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.316599    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:33.735555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.735270    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:33.735555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.735331    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:33.775566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.772213    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:33.775566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.772279    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:33.787555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.786085    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:33.787555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.786141    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:33.791417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.791388    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:33.791658 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.791632    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:33.850682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.850633    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:33.851954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.851261    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 09:45:33.852662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.852641    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:45:33.853331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.853315    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:45:33.882558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.881707    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:33.882558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.882196    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:45:33.904556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.892162    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 09:45:33.904556 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:33.892499    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:45:34.177793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.177704    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:45:34.196763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.196732    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.283651    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.315314    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.315718    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.315919    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.316104    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.316152    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:45:34.355558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.340876    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:34.408557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.407782    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:45:34.782565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.778802    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:34.797659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.790610    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:34.899558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.892385    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.900139    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.908282    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.939849    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.924132    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.923231    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.923178    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.940160    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.924069    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:34.958560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:34.940205    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:35.339558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.315627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:35.339558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.318125    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:35.339558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.318448    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 09:45:35.425560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.397282    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:35.724049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.723934    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:35.724607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.724566    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:35.829564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.822764    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:35.829564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.822834    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:35.829564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.823754    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:35.829564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.823797    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:35.829564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.824092    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:35.829564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.824132    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:35.939807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.939743    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:35.940813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.940776    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:35.963951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.944668    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 09:45:35.964432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:35.944709    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 09:45:36.120559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.119301    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:36.317559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.316628    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 09:45:36.322557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.321634    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 01 09:45:36.322557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.322007    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 01 09:45:36.688560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.686916    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:45:36.749726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.749680    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:36.750032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.749995    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:36.773559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.770963    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:36.773559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.771034    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:36.795557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.795036    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:36.795557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.795098    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:36.796011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.795985    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:36.796196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.796174    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:36.984340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.977188    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:36.985235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.985198    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:36.985469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.985447    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 01 09:45:36.986929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:36.986906    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:37.226911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.212341    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np"
Nov 01 09:45:37.318615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.318298    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 09:45:37.337559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.322661    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:37.337559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.322616    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:37.496567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.483632    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:37.731936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.731824    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:37.731936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.731881    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:37.765018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.764980    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:45:37.769997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.769967    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:37.770254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.770225    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:37.778990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.778870    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:37.778990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.778921    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:37.804560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.791000    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:37.804560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.791053    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:37.808553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.806249    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:45:37.822558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.819581    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:45:37.827006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.826984    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:45:37.839551 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.835831    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:45:37.852968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:37.852939    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:45:38.723267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.723223    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:38.723809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.723292    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:38.770341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.770310    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:38.770595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.770367    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:38.776651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.776623    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:38.776778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.776673    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:38.790566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.790517    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:38.790679 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:38.790581    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:39.723191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.723128    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 01 09:45:39.723418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.723218    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 01 09:45:39.770599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.770555    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Nov 01 09:45:39.770792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.770626    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Nov 01 09:45:39.776691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.776655    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 01 09:45:39.776875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.776734    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 01 09:45:39.791006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.790972    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 01 09:45:39.791166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:39.791032    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 01 09:45:41.345558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.333493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:41.497794 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:45:41.497752    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:45:41.717723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.717617    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:41.775688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.775651    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4x6zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:45:41.775945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.775914    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:41.823671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.823635    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-765gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:45:41.823914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.823883    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:41.835982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.835681    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-tt7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:45:41.835982 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.835725    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:41.836231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.836212    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bn6q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:45:41.836349 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.836330    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:41.894317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.890627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:41.960277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.960236    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:42.003557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:41.987619    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:45:42.040555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.038334    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:42.040555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.039479    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:42.059195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.059171    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:45:42.145114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.145084    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:42.146503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.146452    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:42.147164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.147143    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:45:42.240574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.240475    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:42.241003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.240982    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:42.241793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.241766    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 09:45:42.445515 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.445415    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:42.451702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.451671    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:42.452973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:42.452868    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:45:45.786147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:45.786115    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:47.598935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:47.598894    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:47.777267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:47.777234    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:47.844917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:47.843801    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"]
Nov 01 09:45:47.888279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:47.888208    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f18f-ljsks"]
Nov 01 09:45:48.123280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:48.123243    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:48.930231 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:48.930199    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f899b215-83e5-4a09-842a-9b641615f02d" path="/var/lib/kubelet/pods/f899b215-83e5-4a09-842a-9b641615f02d/volumes"
Nov 01 09:45:50.010006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.009975    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"]
Nov 01 09:45:50.014317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.014297    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.062934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.062891    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:45:50.109146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.109119    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:45:50.109146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.109118    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:45:50.109348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.109167    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:45:50.112626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.112606    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"]
Nov 01 09:45:50.158722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.158690    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.158887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.158811    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.158931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.158905    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh9x\" (UniqueName: \"kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.260796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.260703    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh9x\" (UniqueName: \"kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.260966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.260852    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.260966 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.260929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.261636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.261617    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.263218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.263194    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.318421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.318382    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh9x\" (UniqueName: \"kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x\") pod \"maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.329185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.329158    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:50.369316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.369279    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:50.510871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.510771    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:50.679882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.679817    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:50.707708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.707672    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"]
Nov 01 09:45:50.708481 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:45:50.708452    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a1cb3e_4d0b_4797_bc61_59e758e3f852.slice/crio-207048af34186ee5257820c03d7dd530537cfc09702f5a1f3cf3b6b3fb3a0baf WatchSource:0}: Error finding container 207048af34186ee5257820c03d7dd530537cfc09702f5a1f3cf3b6b3fb3a0baf: Status 404 returned error can't find the container with id 207048af34186ee5257820c03d7dd530537cfc09702f5a1f3cf3b6b3fb3a0baf
Nov 01 09:45:50.833480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:50.833382    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:51.390233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:51.390200    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5" event={"ID":"b5a1cb3e-4d0b-4797-bc61-59e758e3f852","Type":"ContainerStarted","Data":"2d4aea3250eab3d895af8ecb01c51956ad6cdf7b7769fabf5346ef7a8005d6f8"}
Nov 01 09:45:51.390667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:51.390256    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5" event={"ID":"b5a1cb3e-4d0b-4797-bc61-59e758e3f852","Type":"ContainerStarted","Data":"207048af34186ee5257820c03d7dd530537cfc09702f5a1f3cf3b6b3fb3a0baf"}
Nov 01 09:45:51.506474 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:51.506420    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5" podStartSLOduration=2.506406308 podStartE2EDuration="2.506406308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:51.503776302 +0000 UTC m=+4424.698170856" watchObservedRunningTime="2026-11-01 09:45:51.506406308 +0000 UTC m=+4424.700800901"
Nov 01 09:45:52.398601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:52.398572    4314 generic.go:356] "Generic (PLEG): container finished" podID="b5a1cb3e-4d0b-4797-bc61-59e758e3f852" containerID="2d4aea3250eab3d895af8ecb01c51956ad6cdf7b7769fabf5346ef7a8005d6f8" exitCode=0
Nov 01 09:45:52.399029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:52.398639    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5" event={"ID":"b5a1cb3e-4d0b-4797-bc61-59e758e3f852","Type":"ContainerDied","Data":"2d4aea3250eab3d895af8ecb01c51956ad6cdf7b7769fabf5346ef7a8005d6f8"}
Nov 01 09:45:53.136528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.136493    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:45:53.560090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.560064    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:53.709152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.709077    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x\" (UniqueName: \"kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x\") pod \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") "
Nov 01 09:45:53.709308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.709227    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca\") pod \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") "
Nov 01 09:45:53.709308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.709278    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca\") pod \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\" (UID: \"b5a1cb3e-4d0b-4797-bc61-59e758e3f852\") "
Nov 01 09:45:53.709629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.709605    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca" pod "b5a1cb3e-4d0b-4797-bc61-59e758e3f852" (UID: "b5a1cb3e-4d0b-4797-bc61-59e758e3f852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:53.709867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.709848    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:53.711125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.711103    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x" pod "b5a1cb3e-4d0b-4797-bc61-59e758e3f852" (UID: "b5a1cb3e-4d0b-4797-bc61-59e758e3f852"). InnerVolumeSpecName "kube-api-access-fmh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:53.711248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.711227    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca" pod "b5a1cb3e-4d0b-4797-bc61-59e758e3f852" (UID: "b5a1cb3e-4d0b-4797-bc61-59e758e3f852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:45:53.810883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.810853    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh9x\" (UniqueName: \"kubernetes.io/projected/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-kube-api-access-fmh9x\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:53.810883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:53.810881    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1cb3e-4d0b-4797-bc61-59e758e3f852-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:54.414465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:54.414436    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"
Nov 01 09:45:54.414649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:54.414467    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5" event={"ID":"b5a1cb3e-4d0b-4797-bc61-59e758e3f852","Type":"ContainerDied","Data":"207048af34186ee5257820c03d7dd530537cfc09702f5a1f3cf3b6b3fb3a0baf"}
Nov 01 09:45:54.414649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:54.414513    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207048af34186ee5257820c03d7dd530537cfc09702f5a1f3cf3b6b3fb3a0baf"
Nov 01 09:45:58.376922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:45:58.376886    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:03.125940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:03.125911    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:07.872462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:07.872428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:11.394801 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:46:11.394731    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:46:12.250557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.246769    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"]
Nov 01 09:46:12.250557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.247835    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b5a1cb3e-4d0b-4797-bc61-59e758e3f852" containerName="revoke-keys"
Nov 01 09:46:12.250557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.247858    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a1cb3e-4d0b-4797-bc61-59e758e3f852" containerName="revoke-keys"
Nov 01 09:46:12.250557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.248024    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a1cb3e-4d0b-4797-bc61-59e758e3f852" containerName="revoke-keys"
Nov 01 09:46:12.308555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.305793    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"]
Nov 01 09:46:12.308555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.305920    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.389204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.360060    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:46:12.389204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.388810    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.389204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.388940    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.389204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.388995    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.454602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.445756    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:46:12.454602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.445868    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:46:12.454602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.445961    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:46:12.508611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.508320    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.508611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.508402    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.508611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.508570    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.509477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.509448    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.525144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.525086    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.545903 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.545631    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np" podUID="52a83d0f-f6cd-4498-9982-6dae24b6639b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": context deadline exceeded"
Nov 01 09:46:12.612595 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.607381    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9\") pod \"maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:12.828565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:12.828436    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:13.027619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:13.027583    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:13.202013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:13.201985    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"]
Nov 01 09:46:13.203340 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:46:13.203313    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e3c2e6_8839_4563_8163_74779a1b9151.slice/crio-dc64d2839640523b61135c267a40d223a55384b2cf2c88a66690e654d01fe2b8 WatchSource:0}: Error finding container dc64d2839640523b61135c267a40d223a55384b2cf2c88a66690e654d01fe2b8: Status 404 returned error can't find the container with id dc64d2839640523b61135c267a40d223a55384b2cf2c88a66690e654d01fe2b8
Nov 01 09:46:13.642319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:13.642286    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl" event={"ID":"28e3c2e6-8839-4563-8163-74779a1b9151","Type":"ContainerStarted","Data":"6dd0d39def1c51b0c904d41d22146caec7e12c570dee3331c35ad852c6b73997"}
Nov 01 09:46:13.642807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:13.642343    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl" event={"ID":"28e3c2e6-8839-4563-8163-74779a1b9151","Type":"ContainerStarted","Data":"dc64d2839640523b61135c267a40d223a55384b2cf2c88a66690e654d01fe2b8"}
Nov 01 09:46:13.752333 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:13.752275    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl" podStartSLOduration=1.752261178 podStartE2EDuration="1.752261178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:46:13.749693308 +0000 UTC m=+4446.944087864" watchObservedRunningTime="2026-11-01 09:46:13.752261178 +0000 UTC m=+4446.946655732"
Nov 01 09:46:14.650454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:14.650425    4314 generic.go:356] "Generic (PLEG): container finished" podID="28e3c2e6-8839-4563-8163-74779a1b9151" containerID="6dd0d39def1c51b0c904d41d22146caec7e12c570dee3331c35ad852c6b73997" exitCode=0
Nov 01 09:46:14.650864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:14.650508    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl" event={"ID":"28e3c2e6-8839-4563-8163-74779a1b9151","Type":"ContainerDied","Data":"6dd0d39def1c51b0c904d41d22146caec7e12c570dee3331c35ad852c6b73997"}
Nov 01 09:46:15.837007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.836985    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:15.957838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.957761    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca\") pod \"28e3c2e6-8839-4563-8163-74779a1b9151\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") "
Nov 01 09:46:15.958651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.958622    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca\") pod \"28e3c2e6-8839-4563-8163-74779a1b9151\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") "
Nov 01 09:46:15.958897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.958867    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9\") pod \"28e3c2e6-8839-4563-8163-74779a1b9151\" (UID: \"28e3c2e6-8839-4563-8163-74779a1b9151\") "
Nov 01 09:46:15.959036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.958516    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca" pod "28e3c2e6-8839-4563-8163-74779a1b9151" (UID: "28e3c2e6-8839-4563-8163-74779a1b9151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:46:15.960561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.960520    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:46:15.961033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.961010    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca" pod "28e3c2e6-8839-4563-8163-74779a1b9151" (UID: "28e3c2e6-8839-4563-8163-74779a1b9151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:46:15.961333 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:15.961311    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9" pod "28e3c2e6-8839-4563-8163-74779a1b9151" (UID: "28e3c2e6-8839-4563-8163-74779a1b9151"). InnerVolumeSpecName "kube-api-access-jxkk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:46:16.062006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:16.061981    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e3c2e6-8839-4563-8163-74779a1b9151-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:46:16.062006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:16.062007    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/28e3c2e6-8839-4563-8163-74779a1b9151-kube-api-access-jxkk9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:46:16.665613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:16.665580    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl" event={"ID":"28e3c2e6-8839-4563-8163-74779a1b9151","Type":"ContainerDied","Data":"dc64d2839640523b61135c267a40d223a55384b2cf2c88a66690e654d01fe2b8"}
Nov 01 09:46:16.665613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:16.665612    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"
Nov 01 09:46:16.665830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:16.665628    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc64d2839640523b61135c267a40d223a55384b2cf2c88a66690e654d01fe2b8"
Nov 01 09:46:17.870309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:17.870273    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:22.692263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:22.692179    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:25.433082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:25.433053    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:27.478606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:27.478576    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:32.228525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:32.228489    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:36.962984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:36.962954    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:41.322591 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:46:41.322557    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:46:41.467059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.467021    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"]
Nov 01 09:46:41.467794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.467771    4314 cpu_manager.go:414] "containerMap: removing container" podUID="28e3c2e6-8839-4563-8163-74779a1b9151" containerName="revoke-keys"
Nov 01 09:46:41.467794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.467793    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e3c2e6-8839-4563-8163-74779a1b9151" containerName="revoke-keys"
Nov 01 09:46:41.468015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.468001    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e3c2e6-8839-4563-8163-74779a1b9151" containerName="revoke-keys"
Nov 01 09:46:41.471962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.471941    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.493003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.492979    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.493135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.493120    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.493212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.493197    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrkl\" (UniqueName: \"kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.525230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.525200    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"]
Nov 01 09:46:41.565776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.565747    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:46:41.565915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.565803    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:46:41.565915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.565804    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:46:41.566048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.566036    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:46:41.595042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.594966    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.595180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.595109    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.595249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.595186    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrkl\" (UniqueName: \"kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.595737 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.595717    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.597364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.597347    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.651782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.651758    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrkl\" (UniqueName: \"kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl\") pod \"maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.786018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.785985    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:41.923872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:41.923830    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:42.160999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:42.160970    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"]
Nov 01 09:46:42.161867 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:46:42.161842    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4198b9_6a36_4254_811a_2fa2e3f7f10f.slice/crio-e423ef432e08bd8a77c380c4c65ad777df01e60ec4023c9fdfa8b59b0f066240 WatchSource:0}: Error finding container e423ef432e08bd8a77c380c4c65ad777df01e60ec4023c9fdfa8b59b0f066240: Status 404 returned error can't find the container with id e423ef432e08bd8a77c380c4c65ad777df01e60ec4023c9fdfa8b59b0f066240
Nov 01 09:46:42.854007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:42.853970    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk" event={"ID":"ab4198b9-6a36-4254-811a-2fa2e3f7f10f","Type":"ContainerStarted","Data":"041a16242ff36a13cee52df9244bd71247dbcb44aa27f74ab72ee891b034a135"}
Nov 01 09:46:42.854403 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:42.854054    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk" event={"ID":"ab4198b9-6a36-4254-811a-2fa2e3f7f10f","Type":"ContainerStarted","Data":"e423ef432e08bd8a77c380c4c65ad777df01e60ec4023c9fdfa8b59b0f066240"}
Nov 01 09:46:42.964763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:42.964707    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk" podStartSLOduration=1.964688777 podStartE2EDuration="1.964688777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:46:42.961425207 +0000 UTC m=+4476.155819762" watchObservedRunningTime="2026-11-01 09:46:42.964688777 +0000 UTC m=+4476.159083332"
Nov 01 09:46:43.862847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:43.862815    4314 generic.go:356] "Generic (PLEG): container finished" podID="ab4198b9-6a36-4254-811a-2fa2e3f7f10f" containerID="041a16242ff36a13cee52df9244bd71247dbcb44aa27f74ab72ee891b034a135" exitCode=0
Nov 01 09:46:43.863263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:43.862902    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk" event={"ID":"ab4198b9-6a36-4254-811a-2fa2e3f7f10f","Type":"ContainerDied","Data":"041a16242ff36a13cee52df9244bd71247dbcb44aa27f74ab72ee891b034a135"}
Nov 01 09:46:45.032646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.032619    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:45.146321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.146217    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca\") pod \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") "
Nov 01 09:46:45.146507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.146339    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca\") pod \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") "
Nov 01 09:46:45.146620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.146595    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl\" (UniqueName: \"kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl\") pod \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\" (UID: \"ab4198b9-6a36-4254-811a-2fa2e3f7f10f\") "
Nov 01 09:46:45.146901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.146870    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca" pod "ab4198b9-6a36-4254-811a-2fa2e3f7f10f" (UID: "ab4198b9-6a36-4254-811a-2fa2e3f7f10f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:46:45.148194 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.148166    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:46:45.149587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.149520    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca" pod "ab4198b9-6a36-4254-811a-2fa2e3f7f10f" (UID: "ab4198b9-6a36-4254-811a-2fa2e3f7f10f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:46:45.149885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.149861    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl" pod "ab4198b9-6a36-4254-811a-2fa2e3f7f10f" (UID: "ab4198b9-6a36-4254-811a-2fa2e3f7f10f"). InnerVolumeSpecName "kube-api-access-bxrkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:46:45.249044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.249009    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:46:45.249044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.249042    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxrkl\" (UniqueName: \"kubernetes.io/projected/ab4198b9-6a36-4254-811a-2fa2e3f7f10f-kube-api-access-bxrkl\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:46:45.877984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.877952    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"
Nov 01 09:46:45.877984 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.877985    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk" event={"ID":"ab4198b9-6a36-4254-811a-2fa2e3f7f10f","Type":"ContainerDied","Data":"e423ef432e08bd8a77c380c4c65ad777df01e60ec4023c9fdfa8b59b0f066240"}
Nov 01 09:46:45.878233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:45.878064    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e423ef432e08bd8a77c380c4c65ad777df01e60ec4023c9fdfa8b59b0f066240"
Nov 01 09:46:46.632133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:46.632101    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:46.780558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:46.780488    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:46.926907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:46.926786    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:51.550700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:51.550657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:46:56.398785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:46:56.398752    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:01.136259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:01.136221    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:05.886712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:05.886679    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:10.630306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:10.630270    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:11.326246 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:11.326208    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:47:15.395249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:15.395216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:17.675029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:17.674986    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:47:17.675548 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:17.675054    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:47:18.125743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:18.125662    4314 generic.go:356] "Generic (PLEG): container finished" podID="0ada1f59-3446-416f-900c-4e7a192b4399" containerID="abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589" exitCode=0
Nov 01 09:47:18.125940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:18.125740    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" event={"ID":"0ada1f59-3446-416f-900c-4e7a192b4399","Type":"ContainerDied","Data":"abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"}
Nov 01 09:47:18.126204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:18.126183    4314 scope.go:122] "RemoveContainer" containerID="abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"
Nov 01 09:47:18.127892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:18.127860    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:47:18.127976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:18.127936    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:47:18.127976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:18.127971    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:47:19.134188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:19.134153    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" event={"ID":"0ada1f59-3446-416f-900c-4e7a192b4399","Type":"ContainerStarted","Data":"20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741"}
Nov 01 09:47:19.836409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:19.836369    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:47:19.836629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:19.836596    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 01 09:47:19.836744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:19.836655    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 01 09:47:20.188937 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:20.188840    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:20.835953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:20.835914    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 01 09:47:20.836142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:20.835972    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 01 09:47:21.836470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:21.836438    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 01 09:47:21.836900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:21.836489    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 01 09:47:22.835830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:22.835797    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 01 09:47:22.836095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:22.835859    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 01 09:47:23.836179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:23.836144    4314 patch_prober.go:29] interesting pod/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 01 09:47:23.836661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:23.836200    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 01 09:47:24.401816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:24.401777    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:24.840609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:24.840578    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:47:24.841028 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:24.841010    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:47:24.841740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:24.841721    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:47:25.768761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:25.768729    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:26.432883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:26.432845    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"]
Nov 01 09:47:26.478914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:26.478881    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cddc5b-zx5r5"]
Nov 01 09:47:26.653435 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:26.653403    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"]
Nov 01 09:47:26.653698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:26.653656    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" containerID="cri-o://2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b" gracePeriod=30
Nov 01 09:47:26.934458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:26.934426    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a1cb3e-4d0b-4797-bc61-59e758e3f852" path="/var/lib/kubelet/pods/b5a1cb3e-4d0b-4797-bc61-59e758e3f852/volumes"
Nov 01 09:47:27.249723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:27.249693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:27.451476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:27.451442    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:28.631822 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:28.631793    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:28.772663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:28.772633    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:29.631129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:29.631089    4314 patch_prober.go:29] interesting pod/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:47:29.631307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:29.631150    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:47:31.312705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:31.312674    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:31.915081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:31.915057    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:47:32.015912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.015867    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.015950    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016030    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016051    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016076    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016129    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016133    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:32.016326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016204    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016234    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016308    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz\") pod \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\" (UID: \"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7\") "
Nov 01 09:47:32.016661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016483    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:32.016661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016468    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:32.016661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016511    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:32.016661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016594    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:32.016970 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016949    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.017039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016971    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.017039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016982    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.017039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.016992    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.017039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.017002    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.018568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.018509    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:32.018694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.018608    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "kube-api-access-ncjbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:32.018803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.018782    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:47:32.018803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.018792    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token" pod "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" (UID: "c0461a82-08d9-4f66-aa90-c5a5ab48bcf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:32.118180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.118144    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.118180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.118176    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.118180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.118186    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.118429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.118196    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncjbz\" (UniqueName: \"kubernetes.io/projected/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7-kube-api-access-ncjbz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:32.253139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.253100    4314 generic.go:356] "Generic (PLEG): container finished" podID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerID="2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b" exitCode=0
Nov 01 09:47:32.253344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.253168    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"
Nov 01 09:47:32.253344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.253188    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" event={"ID":"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7","Type":"ContainerDied","Data":"2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b"}
Nov 01 09:47:32.253344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.253258    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg" event={"ID":"c0461a82-08d9-4f66-aa90-c5a5ab48bcf7","Type":"ContainerDied","Data":"b7b086019cb478866c80eb997713612d33d78f8ed569bc2b103bec0639a62b7e"}
Nov 01 09:47:32.253344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.253287    4314 scope.go:122] "RemoveContainer" containerID="2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b"
Nov 01 09:47:32.264541 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.264509    4314 scope.go:122] "RemoveContainer" containerID="15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"
Nov 01 09:47:32.274119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.274103    4314 scope.go:122] "RemoveContainer" containerID="2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b"
Nov 01 09:47:32.274376 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:32.274356    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b\": container with ID starting with 2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b not found: ID does not exist" containerID="2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b"
Nov 01 09:47:32.274445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.274384    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b"} err="failed to get container status \"2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b\": rpc error: code = NotFound desc = could not find container \"2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b\": container with ID starting with 2605f076d622a2b22944a750f90b5d0467a3678c1e8ff21e9d9dda6a13fd327b not found: ID does not exist"
Nov 01 09:47:32.274445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.274401    4314 scope.go:122] "RemoveContainer" containerID="15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"
Nov 01 09:47:32.274666 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:32.274648    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3\": container with ID starting with 15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3 not found: ID does not exist" containerID="15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"
Nov 01 09:47:32.274717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.274674    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3"} err="failed to get container status \"15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3\": rpc error: code = NotFound desc = could not find container \"15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3\": container with ID starting with 15b060cb27e92a31fd1c1e3affbe03d02291358237520036428f272f021a6ff3 not found: ID does not exist"
Nov 01 09:47:32.409945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.409907    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"]
Nov 01 09:47:32.413653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.413625    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cddc5b-openshift-default-76d564fb8f-7zjhg"]
Nov 01 09:47:32.929761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:32.929730    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" path="/var/lib/kubelet/pods/c0461a82-08d9-4f66-aa90-c5a5ab48bcf7/volumes"
Nov 01 09:47:33.683102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:33.683064    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:34.217555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:34.217491    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"]
Nov 01 09:47:34.320042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:34.320000    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"]
Nov 01 09:47:34.548999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:34.548948    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:34.694088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:34.694055    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:35.791615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:35.791582    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:37.716203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:37.716167    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:39.218509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.218469    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb" podUID="19919798-47c7-4fb6-9c55-7c42aa189e28" containerName="payload-processing" containerID="cri-o://a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4" gracePeriod=25
Nov 01 09:47:39.321043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.321008    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf" podUID="8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" containerName="payload-pre-processing" containerID="cri-o://77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498" gracePeriod=25
Nov 01 09:47:39.466311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.466285    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:39.506745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.506714    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:47:39.589184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.589160    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:47:39.627388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.627357    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb\" (UniqueName: \"kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb\") pod \"19919798-47c7-4fb6-9c55-7c42aa189e28\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") "
Nov 01 09:47:39.627594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.627465    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume\") pod \"19919798-47c7-4fb6-9c55-7c42aa189e28\" (UID: \"19919798-47c7-4fb6-9c55-7c42aa189e28\") "
Nov 01 09:47:39.628200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.628174    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume" pod "19919798-47c7-4fb6-9c55-7c42aa189e28" (UID: "19919798-47c7-4fb6-9c55-7c42aa189e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:39.629691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.629665    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb" pod "19919798-47c7-4fb6-9c55-7c42aa189e28" (UID: "19919798-47c7-4fb6-9c55-7c42aa189e28"). InnerVolumeSpecName "kube-api-access-bnsjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:39.728776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.728742    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume\") pod \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") "
Nov 01 09:47:39.729068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.729017    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9\" (UniqueName: \"kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9\") pod \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\" (UID: \"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf\") "
Nov 01 09:47:39.729172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.729136    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume" pod "8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" (UID: "8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:39.729630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.729608    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:39.729739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.729633    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnsjb\" (UniqueName: \"kubernetes.io/projected/19919798-47c7-4fb6-9c55-7c42aa189e28-kube-api-access-bnsjb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:39.729739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.729651    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19919798-47c7-4fb6-9c55-7c42aa189e28-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:39.731195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.731168    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9" pod "8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" (UID: "8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf"). InnerVolumeSpecName "kube-api-access-8h2x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:39.831581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:39.831512    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h2x9\" (UniqueName: \"kubernetes.io/projected/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf-kube-api-access-8h2x9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:40.316596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.316561    4314 generic.go:356] "Generic (PLEG): container finished" podID="19919798-47c7-4fb6-9c55-7c42aa189e28" containerID="a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4" exitCode=0
Nov 01 09:47:40.317057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.316618    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"
Nov 01 09:47:40.317057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.316640    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb" event={"ID":"19919798-47c7-4fb6-9c55-7c42aa189e28","Type":"ContainerDied","Data":"a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4"}
Nov 01 09:47:40.317057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.316703    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb" event={"ID":"19919798-47c7-4fb6-9c55-7c42aa189e28","Type":"ContainerDied","Data":"743115c17e8b97f93414928a0a0560a0d180a98ca4e67f9923c6dff8e88f7f1c"}
Nov 01 09:47:40.317057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.316732    4314 scope.go:122] "RemoveContainer" containerID="a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4"
Nov 01 09:47:40.318366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.318346    4314 generic.go:356] "Generic (PLEG): container finished" podID="8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" containerID="77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498" exitCode=0
Nov 01 09:47:40.318484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.318435    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf" event={"ID":"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf","Type":"ContainerDied","Data":"77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498"}
Nov 01 09:47:40.318484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.318474    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf" event={"ID":"8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf","Type":"ContainerDied","Data":"d66d791ac174aa10914d7ba1626929451c3ffe241674823400c6a829709101e7"}
Nov 01 09:47:40.318661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.318590    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"
Nov 01 09:47:40.330651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.330630    4314 scope.go:122] "RemoveContainer" containerID="a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4"
Nov 01 09:47:40.331104 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:40.331075    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4\": container with ID starting with a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4 not found: ID does not exist" containerID="a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4"
Nov 01 09:47:40.331214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.331115    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4"} err="failed to get container status \"a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4\": rpc error: code = NotFound desc = could not find container \"a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4\": container with ID starting with a7862315dfe0dcfb79e315eca284381d643a3fbbed35324920df31d5dcf2f4b4 not found: ID does not exist"
Nov 01 09:47:40.331214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.331142    4314 scope.go:122] "RemoveContainer" containerID="77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498"
Nov 01 09:47:40.340480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.340464    4314 scope.go:122] "RemoveContainer" containerID="77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498"
Nov 01 09:47:40.340758 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:40.340740    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498\": container with ID starting with 77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498 not found: ID does not exist" containerID="77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498"
Nov 01 09:47:40.340845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.340762    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498"} err="failed to get container status \"77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498\": rpc error: code = NotFound desc = could not find container \"77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498\": container with ID starting with 77c13ff9353751e6401bda47a46feecc889800094bfee602a33a882c76fbe498 not found: ID does not exist"
Nov 01 09:47:40.471326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.471295    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"]
Nov 01 09:47:40.476599 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.476576    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c81a9d-7cf775fc67-4ngnf"]
Nov 01 09:47:40.578297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.578225    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"]
Nov 01 09:47:40.585010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.584985    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c81a9d-5cd9688949-p9fcb"]
Nov 01 09:47:40.929889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.929815    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19919798-47c7-4fb6-9c55-7c42aa189e28" path="/var/lib/kubelet/pods/19919798-47c7-4fb6-9c55-7c42aa189e28/volumes"
Nov 01 09:47:40.930232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:40.930218    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" path="/var/lib/kubelet/pods/8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf/volumes"
Nov 01 09:47:41.330375 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:41.330343    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:47:42.524586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:42.524526    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:46.661991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:46.661960    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:48.355013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.354981    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:48.497243 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.497198    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:48.574526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.574494    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"]
Nov 01 09:47:48.620049 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.619970    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a53f2-rfjvl"]
Nov 01 09:47:48.797044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.796998    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"]
Nov 01 09:47:48.797268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.797242    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy" containerID="cri-o://ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956" gracePeriod=30
Nov 01 09:47:48.829056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.828985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:48.970084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:48.970003    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e3c2e6-8839-4563-8163-74779a1b9151" path="/var/lib/kubelet/pods/28e3c2e6-8839-4563-8163-74779a1b9151/volumes"
Nov 01 09:47:49.570810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:49.570778    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:49.716957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:49.716922    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:53.604921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:53.604889    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:54.085050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.085020    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:47:54.155229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155195    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155235    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155275    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155355    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155388    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155451    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155635    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155726    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:54.155783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155752    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.155961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155830    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket\") pod \"dce0b826-d172-4dc7-9303-d1397880cf52\" (UID: \"dce0b826-d172-4dc7-9303-d1397880cf52\") "
Nov 01 09:47:54.156033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155751    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:54.156033 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.155962    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:54.156160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156031    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:54.156228 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156188    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:54.156482 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156460    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.156609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156487    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.156609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156502    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.156609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156520    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce0b826-d172-4dc7-9303-d1397880cf52-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.156609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.156553    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.157757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.157731    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:47:54.157927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.157909    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "kube-api-access-8bkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:54.158053 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.158034    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:47:54.158099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.158053    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token" pod "dce0b826-d172-4dc7-9303-d1397880cf52" (UID: "dce0b826-d172-4dc7-9303-d1397880cf52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:54.258034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.257994    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce0b826-d172-4dc7-9303-d1397880cf52-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.258034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.258033    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce0b826-d172-4dc7-9303-d1397880cf52-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.258328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.258049    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.258328 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.258065    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bkv8\" (UniqueName: \"kubernetes.io/projected/dce0b826-d172-4dc7-9303-d1397880cf52-kube-api-access-8bkv8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:54.421686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.421605    4314 generic.go:356] "Generic (PLEG): container finished" podID="dce0b826-d172-4dc7-9303-d1397880cf52" containerID="ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956" exitCode=0
Nov 01 09:47:54.421686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.421667    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"
Nov 01 09:47:54.421882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.421698    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" event={"ID":"dce0b826-d172-4dc7-9303-d1397880cf52","Type":"ContainerDied","Data":"ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956"}
Nov 01 09:47:54.421882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.421759    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4" event={"ID":"dce0b826-d172-4dc7-9303-d1397880cf52","Type":"ContainerDied","Data":"9076374d7b5d95a96973e7b8dff8bd1f323728908e83c8017aca88a6b0fea693"}
Nov 01 09:47:54.421882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.421786    4314 scope.go:122] "RemoveContainer" containerID="ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956"
Nov 01 09:47:54.434879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.434859    4314 scope.go:122] "RemoveContainer" containerID="4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"
Nov 01 09:47:54.444264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.444248    4314 scope.go:122] "RemoveContainer" containerID="ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956"
Nov 01 09:47:54.444558 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:54.444511    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956\": container with ID starting with ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956 not found: ID does not exist" containerID="ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956"
Nov 01 09:47:54.444633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.444556    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956"} err="failed to get container status \"ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956\": rpc error: code = NotFound desc = could not find container \"ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956\": container with ID starting with ea1fa6744affe3c6f51b4413b75f5aa76361600af43e7191051f36eaa04db956 not found: ID does not exist"
Nov 01 09:47:54.444633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.444578    4314 scope.go:122] "RemoveContainer" containerID="4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"
Nov 01 09:47:54.444814 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:47:54.444799    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114\": container with ID starting with 4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114 not found: ID does not exist" containerID="4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"
Nov 01 09:47:54.444861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.444819    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114"} err="failed to get container status \"4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114\": rpc error: code = NotFound desc = could not find container \"4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114\": container with ID starting with 4bb33042253fc522e1a9374b9ff3dc861f3151035c993e00daae321ae9739114 not found: ID does not exist"
Nov 01 09:47:54.576473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.576437    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"]
Nov 01 09:47:54.581981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.581961    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3a53f2-openshift-default-5d65fbf5f-k48n4"]
Nov 01 09:47:54.928969 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:54.928920    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" path="/var/lib/kubelet/pods/dce0b826-d172-4dc7-9303-d1397880cf52/volumes"
Nov 01 09:47:55.674085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:55.674045    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:56.653589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:56.653549    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"]
Nov 01 09:47:57.016361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:57.016325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:57.159266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:57.159235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:47:58.624680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:47:58.624650    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:01.323587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:01.323515    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:01.654559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:01.654448    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4" podUID="c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" containerName="payload-processing" containerID="cri-o://342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475" gracePeriod=25
Nov 01 09:48:01.925728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:01.925702    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:48:02.059606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.059575    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume\") pod \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") "
Nov 01 09:48:02.059766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.059747    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr\" (UniqueName: \"kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr\") pod \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\" (UID: \"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65\") "
Nov 01 09:48:02.059991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.059971    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume" pod "c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" (UID: "c25b77bc-54d3-49af-9f2c-a6bedbe8bd65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:48:02.060662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.060638    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:02.062433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.062406    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr" pod "c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" (UID: "c25b77bc-54d3-49af-9f2c-a6bedbe8bd65"). InnerVolumeSpecName "kube-api-access-66wdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:02.162091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.162017    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wdr\" (UniqueName: \"kubernetes.io/projected/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65-kube-api-access-66wdr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:02.483365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.483333    4314 generic.go:356] "Generic (PLEG): container finished" podID="c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" containerID="342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475" exitCode=0
Nov 01 09:48:02.483895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.483380    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"
Nov 01 09:48:02.483895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.483405    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4" event={"ID":"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65","Type":"ContainerDied","Data":"342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475"}
Nov 01 09:48:02.483895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.483472    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4" event={"ID":"c25b77bc-54d3-49af-9f2c-a6bedbe8bd65","Type":"ContainerDied","Data":"2bf0c1c94347ae6f0c8efc0f41d99d01bdcede5d08c1e269389337b6c6466adc"}
Nov 01 09:48:02.483895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.483503    4314 scope.go:122] "RemoveContainer" containerID="342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475"
Nov 01 09:48:02.494655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.494634    4314 scope.go:122] "RemoveContainer" containerID="342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475"
Nov 01 09:48:02.494923 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:02.494905    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475\": container with ID starting with 342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475 not found: ID does not exist" containerID="342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475"
Nov 01 09:48:02.494995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.494930    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475"} err="failed to get container status \"342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475\": rpc error: code = NotFound desc = could not find container \"342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475\": container with ID starting with 342acfd57561acd1a2c4c67772ee7ccdd0b21f2425eb6480838ddfa759e19475 not found: ID does not exist"
Nov 01 09:48:02.637894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.637864    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"]
Nov 01 09:48:02.685254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.685221    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3d6bb-6fc6dfb45-8m9w4"]
Nov 01 09:48:02.929150 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:02.929060    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" path="/var/lib/kubelet/pods/c25b77bc-54d3-49af-9f2c-a6bedbe8bd65/volumes"
Nov 01 09:48:03.492094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:03.492063    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:08.293326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:08.293291    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:08.447624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:08.447585    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:11.331665 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:11.331619    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:48:12.477716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:12.477682    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:14.232652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.232622    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:14.400175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.400144    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:14.450361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.450331    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"]
Nov 01 09:48:14.495915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.495844    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12a01a-4rzsk"]
Nov 01 09:48:14.672509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.672465    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"]
Nov 01 09:48:14.672839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.672805    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" containerID="cri-o://43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715" gracePeriod=30
Nov 01 09:48:14.724957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.724928    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:14.929733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:14.929652    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4198b9-6a36-4254-811a-2fa2e3f7f10f" path="/var/lib/kubelet/pods/ab4198b9-6a36-4254-811a-2fa2e3f7f10f/volumes"
Nov 01 09:48:15.483546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:15.483510    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:16.783873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:16.783831    4314 patch_prober.go:29] interesting pod/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:48:16.784310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:16.783900    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:48:19.375719 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:19.375686    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:19.954972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:19.954895    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:48:19.973389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:19.973353    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:20.002495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002462    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.002712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002543    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.002712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002571    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.002712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002646    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.002712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002674    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.002712 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002695    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.003010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002741    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.003010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002815    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.003010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002872    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy\") pod \"fbbea89c-74ff-4d50-b5c7-8185927cda88\" (UID: \"fbbea89c-74ff-4d50-b5c7-8185927cda88\") "
Nov 01 09:48:20.003010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.002939    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:20.003303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003073    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:20.003303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003143    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:20.003303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003159    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:20.003303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003088    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:48:20.003555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003523    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.003632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003566    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.003632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003581    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.003632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003599    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.003632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.003613    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbea89c-74ff-4d50-b5c7-8185927cda88-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.004993 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.004966    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:20.005122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.005104    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "kube-api-access-mvxkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:20.005191 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.005175    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:48:20.005259 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.005246    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy" pod "fbbea89c-74ff-4d50-b5c7-8185927cda88" (UID: "fbbea89c-74ff-4d50-b5c7-8185927cda88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:20.104823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.104786    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.104823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.104819    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxkj\" (UniqueName: \"kubernetes.io/projected/fbbea89c-74ff-4d50-b5c7-8185927cda88-kube-api-access-mvxkj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.104823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.104830    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.105087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.104839    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbea89c-74ff-4d50-b5c7-8185927cda88-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:20.116793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.116767    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:20.606850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.606811    4314 generic.go:356] "Generic (PLEG): container finished" podID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerID="43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715" exitCode=0
Nov 01 09:48:20.607327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.606865    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"
Nov 01 09:48:20.607327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.606893    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" event={"ID":"fbbea89c-74ff-4d50-b5c7-8185927cda88","Type":"ContainerDied","Data":"43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715"}
Nov 01 09:48:20.607327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.606951    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6" event={"ID":"fbbea89c-74ff-4d50-b5c7-8185927cda88","Type":"ContainerDied","Data":"c9c7976797c042f7557835f775e3cee7a83ae4282709e3b4e6bdb1bc8d723bde"}
Nov 01 09:48:20.607327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.606975    4314 scope.go:122] "RemoveContainer" containerID="43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715"
Nov 01 09:48:20.618813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.618794    4314 scope.go:122] "RemoveContainer" containerID="81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"
Nov 01 09:48:20.628696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.628675    4314 scope.go:122] "RemoveContainer" containerID="43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715"
Nov 01 09:48:20.629040 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:20.629014    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715\": container with ID starting with 43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715 not found: ID does not exist" containerID="43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715"
Nov 01 09:48:20.629083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.629049    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715"} err="failed to get container status \"43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715\": rpc error: code = NotFound desc = could not find container \"43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715\": container with ID starting with 43fbd974176311eb3d0ece1c324ec8e3e7d2586b848212a6e693402e29ec0715 not found: ID does not exist"
Nov 01 09:48:20.629083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.629070    4314 scope.go:122] "RemoveContainer" containerID="81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"
Nov 01 09:48:20.629337 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:20.629318    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e\": container with ID starting with 81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e not found: ID does not exist" containerID="81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"
Nov 01 09:48:20.629381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.629342    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e"} err="failed to get container status \"81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e\": rpc error: code = NotFound desc = could not find container \"81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e\": container with ID starting with 81b3f2f2c312d63ca7cc5db33af28c06404c27ccf42873327121bf8cd186537e not found: ID does not exist"
Nov 01 09:48:20.762102 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.762070    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"]
Nov 01 09:48:20.808115 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.808074    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-12a01a-openshift-default-8476c99d67-tlpn6"]
Nov 01 09:48:20.929081 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:20.928999    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" path="/var/lib/kubelet/pods/fbbea89c-74ff-4d50-b5c7-8185927cda88/volumes"
Nov 01 09:48:21.769344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:21.769311    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:21.911586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:21.911523    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:23.950574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:23.950517    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:24.095024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:24.094982    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:24.711874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:24.711826    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:26.114378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:26.114344    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:27.777495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:27.777437    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:28.131604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:28.131510    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:28.975185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:28.975139    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"]
Nov 01 09:48:29.073527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:29.073492    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"]
Nov 01 09:48:29.338479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:29.338446    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:29.482326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:29.482291    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:31.251524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:31.251489    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:33.976266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:33.976225    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" podUID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" containerName="payload-processing" containerID="cri-o://02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0" gracePeriod=25
Nov 01 09:48:34.030964 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:48:34.030925    4314 logging.go:55] [core] [Channel #13593 SubChannel #13594]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.70:50052", ServerName: "10.132.1.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.70:50052: connect: connection refused"
Nov 01 09:48:34.074168 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.074122    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6" podUID="d1638a63-b184-4a57-b409-fdb0ca0b6689" containerName="payload-pre-processing" containerID="cri-o://6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db" gracePeriod=25
Nov 01 09:48:34.112867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.112837    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:34.242011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.241983    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:48:34.258865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.258823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:34.295967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.295934    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9\" (UniqueName: \"kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9\") pod \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") "
Nov 01 09:48:34.296160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.296037    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume\") pod \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\" (UID: \"0ea85d98-ce91-4bd7-b225-1cc0a8553c72\") "
Nov 01 09:48:34.296633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.296594    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume" pod "0ea85d98-ce91-4bd7-b225-1cc0a8553c72" (UID: "0ea85d98-ce91-4bd7-b225-1cc0a8553c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:48:34.298055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.298027    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9" pod "0ea85d98-ce91-4bd7-b225-1cc0a8553c72" (UID: "0ea85d98-ce91-4bd7-b225-1cc0a8553c72"). InnerVolumeSpecName "kube-api-access-v5nl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:34.341914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.341887    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:48:34.397519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.397481    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume\") pod \"d1638a63-b184-4a57-b409-fdb0ca0b6689\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") "
Nov 01 09:48:34.397788 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.397636    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d\" (UniqueName: \"kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d\") pod \"d1638a63-b184-4a57-b409-fdb0ca0b6689\" (UID: \"d1638a63-b184-4a57-b409-fdb0ca0b6689\") "
Nov 01 09:48:34.397897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.397873    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume" pod "d1638a63-b184-4a57-b409-fdb0ca0b6689" (UID: "d1638a63-b184-4a57-b409-fdb0ca0b6689"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:48:34.398138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.398124    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5nl9\" (UniqueName: \"kubernetes.io/projected/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-kube-api-access-v5nl9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:34.398180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.398141    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ea85d98-ce91-4bd7-b225-1cc0a8553c72-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:34.398180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.398150    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1638a63-b184-4a57-b409-fdb0ca0b6689-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:34.399702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.399677    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d" pod "d1638a63-b184-4a57-b409-fdb0ca0b6689" (UID: "d1638a63-b184-4a57-b409-fdb0ca0b6689"). InnerVolumeSpecName "kube-api-access-sbg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:34.500030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.499938    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbg4d\" (UniqueName: \"kubernetes.io/projected/d1638a63-b184-4a57-b409-fdb0ca0b6689-kube-api-access-sbg4d\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:34.728295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.728251    4314 generic.go:356] "Generic (PLEG): container finished" podID="d1638a63-b184-4a57-b409-fdb0ca0b6689" containerID="6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db" exitCode=0
Nov 01 09:48:34.728476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.728313    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"
Nov 01 09:48:34.728476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.728341    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6" event={"ID":"d1638a63-b184-4a57-b409-fdb0ca0b6689","Type":"ContainerDied","Data":"6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db"}
Nov 01 09:48:34.728476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.728395    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6" event={"ID":"d1638a63-b184-4a57-b409-fdb0ca0b6689","Type":"ContainerDied","Data":"8efc5e11e7042f9b8ec6dfd8f6e291b01b769dbc83376fc97efd8cda8a7f79d6"}
Nov 01 09:48:34.728476 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.728421    4314 scope.go:122] "RemoveContainer" containerID="6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db"
Nov 01 09:48:34.730210 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.730181    4314 generic.go:356] "Generic (PLEG): container finished" podID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" containerID="02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0" exitCode=0
Nov 01 09:48:34.730341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.730212    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" event={"ID":"0ea85d98-ce91-4bd7-b225-1cc0a8553c72","Type":"ContainerDied","Data":"02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0"}
Nov 01 09:48:34.730341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.730234    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"
Nov 01 09:48:34.730341 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.730274    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" event={"ID":"0ea85d98-ce91-4bd7-b225-1cc0a8553c72","Type":"ContainerDied","Data":"6361aa1808e235cbcd93767eaa34efaf5ba439f7a9fd7d34688fdf3227a43637"}
Nov 01 09:48:34.738844 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.738822    4314 scope.go:122] "RemoveContainer" containerID="6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db"
Nov 01 09:48:34.739208 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:34.739184    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db\": container with ID starting with 6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db not found: ID does not exist" containerID="6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db"
Nov 01 09:48:34.739350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.739226    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db"} err="failed to get container status \"6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db\": rpc error: code = NotFound desc = could not find container \"6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db\": container with ID starting with 6e3ab8f3f846fe8d14f4c2751b348345542fd3389180e03bb3faf1016014b3db not found: ID does not exist"
Nov 01 09:48:34.739350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.739252    4314 scope.go:122] "RemoveContainer" containerID="02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0"
Nov 01 09:48:34.749013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.748987    4314 scope.go:122] "RemoveContainer" containerID="02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0"
Nov 01 09:48:34.749334 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:34.749314    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0\": container with ID starting with 02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0 not found: ID does not exist" containerID="02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0"
Nov 01 09:48:34.749401 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.749341    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0"} err="failed to get container status \"02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0\": rpc error: code = NotFound desc = could not find container \"02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0\": container with ID starting with 02a6151a255b3cc84768ecf649c7d88282f15bee7ded1f6a11e11560297ecbf0 not found: ID does not exist"
Nov 01 09:48:34.882338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.882242    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"]
Nov 01 09:48:34.929900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:34.929867    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g"]
Nov 01 09:48:35.030050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:35.029996    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"]
Nov 01 09:48:35.031221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:35.031184    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3710de-6d4dbc9bc7-kqr7g" podUID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.70:50052\" within 1s: context deadline exceeded"
Nov 01 09:48:35.031362 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:48:35.031333    4314 logging.go:55] [core] [Channel #13593 SubChannel #13594]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.70:50052", ServerName: "10.132.1.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.70:50052: operation was canceled"
Nov 01 09:48:35.033777 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:35.033752    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3710de-7bd7759-pl2k6"]
Nov 01 09:48:35.689398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:35.689364    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:36.168176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:36.168134    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"]
Nov 01 09:48:36.168684 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:36.168355    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" containerID="cri-o://137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43" gracePeriod=30
Nov 01 09:48:36.328983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:36.328049    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:36.471527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:36.471425    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:36.929023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:36.928981    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" path="/var/lib/kubelet/pods/0ea85d98-ce91-4bd7-b225-1cc0a8553c72/volumes"
Nov 01 09:48:36.929590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:36.929573    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1638a63-b184-4a57-b409-fdb0ca0b6689" path="/var/lib/kubelet/pods/d1638a63-b184-4a57-b409-fdb0ca0b6689/volumes"
Nov 01 09:48:37.144830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:37.144787    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:37.305095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:37.305047    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:40.557624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:40.557577    4314 patch_prober.go:29] interesting pod/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:48:40.558141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:40.557644    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:48:41.335124 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:41.334570    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:48:41.450052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.449988    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:48:41.510766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.510729    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:41.597382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597340    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597429    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597466    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597495    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597621    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597650    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597713    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597744    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597748    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597762    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:41.597906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597811    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert\") pod \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\" (UID: \"77cc2eae-cfd1-4440-af18-46721dd4f8ad\") "
Nov 01 09:48:41.598440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.597933    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:41.598440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598244    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:41.598440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598296    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:48:41.598635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598478    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.598635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598502    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.598635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598517    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.598635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598546    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.598635 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.598563    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.599917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.599890    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:41.600064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.599951    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "kube-api-access-kpt8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:41.600122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.600111    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:48:41.600199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.600183    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo" pod "77cc2eae-cfd1-4440-af18-46721dd4f8ad" (UID: "77cc2eae-cfd1-4440-af18-46721dd4f8ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:48:41.699516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.699482    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpt8h\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-kube-api-access-kpt8h\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.699516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.699514    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.699516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.699524    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.699784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.699563    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cc2eae-cfd1-4440-af18-46721dd4f8ad-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:41.798182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.798147    4314 generic.go:356] "Generic (PLEG): container finished" podID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerID="137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43" exitCode=0
Nov 01 09:48:41.798390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.798198    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" event={"ID":"77cc2eae-cfd1-4440-af18-46721dd4f8ad","Type":"ContainerDied","Data":"137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43"}
Nov 01 09:48:41.798390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.798212    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"
Nov 01 09:48:41.798390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.798266    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc" event={"ID":"77cc2eae-cfd1-4440-af18-46721dd4f8ad","Type":"ContainerDied","Data":"ff83ab7aa3c3e9160597e97b132f21a12254d82bcbfe35853cb6645f25520069"}
Nov 01 09:48:41.798390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.798289    4314 scope.go:122] "RemoveContainer" containerID="137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43"
Nov 01 09:48:41.810311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.810290    4314 scope.go:122] "RemoveContainer" containerID="04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"
Nov 01 09:48:41.821483 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.821466    4314 scope.go:122] "RemoveContainer" containerID="137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43"
Nov 01 09:48:41.821815 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:41.821794    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43\": container with ID starting with 137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43 not found: ID does not exist" containerID="137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43"
Nov 01 09:48:41.821902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.821829    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43"} err="failed to get container status \"137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43\": rpc error: code = NotFound desc = could not find container \"137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43\": container with ID starting with 137e0a9a5304481543af755e592a7263b307c4c9c41a55241f9787a019c8df43 not found: ID does not exist"
Nov 01 09:48:41.821902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.821856    4314 scope.go:122] "RemoveContainer" containerID="04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"
Nov 01 09:48:41.822110 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:48:41.822083    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e\": container with ID starting with 04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e not found: ID does not exist" containerID="04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"
Nov 01 09:48:41.822216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.822114    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e"} err="failed to get container status \"04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e\": rpc error: code = NotFound desc = could not find container \"04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e\": container with ID starting with 04625d8a0368a749ea88d1b67c3b00902b7a21cf0b7ab67f0c4f4d2a19670d7e not found: ID does not exist"
Nov 01 09:48:41.956744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.956706    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"]
Nov 01 09:48:41.962872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:41.962838    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f41050-openshift-default-6c9678499c-cxnwc"]
Nov 01 09:48:42.928790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:42.928758    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" path="/var/lib/kubelet/pods/77cc2eae-cfd1-4440-af18-46721dd4f8ad/volumes"
Nov 01 09:48:46.790862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:46.790828    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:51.932985 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:51.932953    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:53.854693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:53.854659    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:57.125285 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.125242    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:57.472236 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472161    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"]
Nov 01 09:48:57.472767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472752    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472768    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472778    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d1638a63-b184-4a57-b409-fdb0ca0b6689" containerName="payload-pre-processing"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472783    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1638a63-b184-4a57-b409-fdb0ca0b6689" containerName="payload-pre-processing"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472800    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472805    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472813    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ab4198b9-6a36-4254-811a-2fa2e3f7f10f" containerName="revoke-keys"
Nov 01 09:48:57.472824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472818    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4198b9-6a36-4254-811a-2fa2e3f7f10f" containerName="revoke-keys"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472833    4314 cpu_manager.go:414] "containerMap: removing container" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472838    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472849    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472854    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472862    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472867    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472877    4314 cpu_manager.go:414] "containerMap: removing container" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472882    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472890    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472894    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472901    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472905    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472918    4314 cpu_manager.go:414] "containerMap: removing container" podUID="19919798-47c7-4fb6-9c55-7c42aa189e28" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472922    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19919798-47c7-4fb6-9c55-7c42aa189e28" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472939    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" containerName="payload-pre-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.472943    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" containerName="payload-pre-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473018    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4198b9-6a36-4254-811a-2fa2e3f7f10f" containerName="revoke-keys"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473029    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473035    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea85d98-ce91-4bd7-b225-1cc0a8553c72" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473041    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473048    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cc2eae-cfd1-4440-af18-46721dd4f8ad" containerName="istio-proxy"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473054    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25b77bc-54d3-49af-9f2c-a6bedbe8bd65" containerName="payload-processing"
Nov 01 09:48:57.473052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473059    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce0b826-d172-4dc7-9303-d1397880cf52" containerName="istio-proxy"
Nov 01 09:48:57.473799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473068    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="19919798-47c7-4fb6-9c55-7c42aa189e28" containerName="payload-processing"
Nov 01 09:48:57.473799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473076    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy"
Nov 01 09:48:57.473799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473082    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy"
Nov 01 09:48:57.473799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473087    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec4d3bd-19de-4fd6-a51c-b09baf06d2bf" containerName="payload-pre-processing"
Nov 01 09:48:57.473799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473092    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy"
Nov 01 09:48:57.473799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.473097    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1638a63-b184-4a57-b409-fdb0ca0b6689" containerName="payload-pre-processing"
Nov 01 09:48:57.476491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.476472    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.530885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.530852    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"]
Nov 01 09:48:57.571688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.571650    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:48:57.571858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.571726    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:48:57.571858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.571754    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:48:57.571858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.571780    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:48:57.577450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.577423    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.577618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.577601    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8r2z\" (UniqueName: \"kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.577664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.577650    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.678611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.678523    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8r2z\" (UniqueName: \"kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.678828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.678668    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.678828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.678767    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.679386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.679361    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.681273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.681251    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.735864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.735832    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8r2z\" (UniqueName: \"kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z\") pod \"maas-api-revoke-sub-c8b52424-749cr\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:57.789492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:57.789448    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:48:58.177252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:58.177227    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"]
Nov 01 09:48:58.179360 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:48:58.179330    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74a67a2_000d_4501_a035_462199f9b896.slice/crio-6116f1e9e335104e229d9ac1726889968b219461a7e5476aeeec17de06b534c7 WatchSource:0}: Error finding container 6116f1e9e335104e229d9ac1726889968b219461a7e5476aeeec17de06b534c7: Status 404 returned error can't find the container with id 6116f1e9e335104e229d9ac1726889968b219461a7e5476aeeec17de06b534c7
Nov 01 09:48:58.917606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:58.917568    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr" event={"ID":"b74a67a2-000d-4501-a035-462199f9b896","Type":"ContainerStarted","Data":"37043857a7ead30de9cb8e84e0075b3fb3028aa1113ae2b288a402d0606369fd"}
Nov 01 09:48:58.917804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:58.917626    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr" event={"ID":"b74a67a2-000d-4501-a035-462199f9b896","Type":"ContainerStarted","Data":"6116f1e9e335104e229d9ac1726889968b219461a7e5476aeeec17de06b534c7"}
Nov 01 09:48:59.026864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:59.026814    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr" podStartSLOduration=2.02680088 podStartE2EDuration="2.02680088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:48:59.025033651 +0000 UTC m=+4612.219428221" watchObservedRunningTime="2026-11-01 09:48:59.02680088 +0000 UTC m=+4612.221195435"
Nov 01 09:48:59.728161 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:59.728123    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:48:59.926291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:59.926256    4314 generic.go:356] "Generic (PLEG): container finished" podID="b74a67a2-000d-4501-a035-462199f9b896" containerID="37043857a7ead30de9cb8e84e0075b3fb3028aa1113ae2b288a402d0606369fd" exitCode=0
Nov 01 09:48:59.926484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:48:59.926330    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr" event={"ID":"b74a67a2-000d-4501-a035-462199f9b896","Type":"ContainerDied","Data":"37043857a7ead30de9cb8e84e0075b3fb3028aa1113ae2b288a402d0606369fd"}
Nov 01 09:49:01.088578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.088517    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:49:01.121590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.121525    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca\") pod \"b74a67a2-000d-4501-a035-462199f9b896\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") "
Nov 01 09:49:01.121759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.121723    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca\") pod \"b74a67a2-000d-4501-a035-462199f9b896\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") "
Nov 01 09:49:01.121759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.121747    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z\" (UniqueName: \"kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z\") pod \"b74a67a2-000d-4501-a035-462199f9b896\" (UID: \"b74a67a2-000d-4501-a035-462199f9b896\") "
Nov 01 09:49:01.122095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.122057    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca" pod "b74a67a2-000d-4501-a035-462199f9b896" (UID: "b74a67a2-000d-4501-a035-462199f9b896"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:49:01.122220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.122186    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74a67a2-000d-4501-a035-462199f9b896-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:01.123662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.123641    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca" pod "b74a67a2-000d-4501-a035-462199f9b896" (UID: "b74a67a2-000d-4501-a035-462199f9b896"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:49:01.123886 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.123867    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z" pod "b74a67a2-000d-4501-a035-462199f9b896" (UID: "b74a67a2-000d-4501-a035-462199f9b896"). InnerVolumeSpecName "kube-api-access-t8r2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:49:01.223306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.223212    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74a67a2-000d-4501-a035-462199f9b896-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:01.223306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.223240    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8r2z\" (UniqueName: \"kubernetes.io/projected/b74a67a2-000d-4501-a035-462199f9b896-kube-api-access-t8r2z\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:01.940555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.940495    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"
Nov 01 09:49:01.940555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.940512    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr" event={"ID":"b74a67a2-000d-4501-a035-462199f9b896","Type":"ContainerDied","Data":"6116f1e9e335104e229d9ac1726889968b219461a7e5476aeeec17de06b534c7"}
Nov 01 09:49:01.940782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:01.940598    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6116f1e9e335104e229d9ac1726889968b219461a7e5476aeeec17de06b534c7"
Nov 01 09:49:03.475659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:03.475619    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:03.618809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:03.618774    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:03.938299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:03.938264    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:04.921439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:04.921405    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:05.065857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:05.065819    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:05.211405 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:05.211322    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:06.232564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:06.232495    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:06.388763 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:06.388731    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:08.181859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.181825    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:08.409489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.409452    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"]
Nov 01 09:49:08.410050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410036    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b74a67a2-000d-4501-a035-462199f9b896" containerName="revoke-keys"
Nov 01 09:49:08.410100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410051    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74a67a2-000d-4501-a035-462199f9b896" containerName="revoke-keys"
Nov 01 09:49:08.410100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410066    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy"
Nov 01 09:49:08.410100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410071    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbea89c-74ff-4d50-b5c7-8185927cda88" containerName="istio-proxy"
Nov 01 09:49:08.410100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410080    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy"
Nov 01 09:49:08.410100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410085    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy"
Nov 01 09:49:08.410274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410162    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74a67a2-000d-4501-a035-462199f9b896" containerName="revoke-keys"
Nov 01 09:49:08.410274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.410171    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0461a82-08d9-4f66-aa90-c5a5ab48bcf7" containerName="istio-proxy"
Nov 01 09:49:08.413830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.413813    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.468393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.468319    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"]
Nov 01 09:49:08.509525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.509489    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:49:08.509525 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.509509    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:49:08.509816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.509656    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:49:08.509816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.509729    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:49:08.516113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.516087    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.516309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.516280    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4lh\" (UniqueName: \"kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.516439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.516420    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.617856 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.617817    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.618065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.617885    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.618065 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.617981    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4lh\" (UniqueName: \"kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.618577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.618519    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.620270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.620242    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.675440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.675409    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4lh\" (UniqueName: \"kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh\") pod \"maas-api-revoke-sub-e2394826-s999x\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.727484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.727410    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:08.876158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:08.876122    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:09.097160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:09.097134    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"]
Nov 01 09:49:09.098548 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:49:09.098505    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10fdfa8_bc02_4e48_8673_9fd121d1222d.slice/crio-0dbc71ec12ae4ec7df50d94affd0b82b1a6ddc0922da1923b8b620df7fd66ab5 WatchSource:0}: Error finding container 0dbc71ec12ae4ec7df50d94affd0b82b1a6ddc0922da1923b8b620df7fd66ab5: Status 404 returned error can't find the container with id 0dbc71ec12ae4ec7df50d94affd0b82b1a6ddc0922da1923b8b620df7fd66ab5
Nov 01 09:49:09.996651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:09.996617    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x" event={"ID":"a10fdfa8-bc02-4e48-8673-9fd121d1222d","Type":"ContainerStarted","Data":"cfa68f01f0249a7c7e1eacc7bedab4197761dcc9e04834d8dfad5222b19a53c4"}
Nov 01 09:49:09.997067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:09.996668    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x" event={"ID":"a10fdfa8-bc02-4e48-8673-9fd121d1222d","Type":"ContainerStarted","Data":"0dbc71ec12ae4ec7df50d94affd0b82b1a6ddc0922da1923b8b620df7fd66ab5"}
Nov 01 09:49:10.107027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:10.106972    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x" podStartSLOduration=2.106957938 podStartE2EDuration="2.106957938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:49:10.104065001 +0000 UTC m=+4623.298459582" watchObservedRunningTime="2026-11-01 09:49:10.106957938 +0000 UTC m=+4623.301352532"
Nov 01 09:49:11.005216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:11.005182    4314 generic.go:356] "Generic (PLEG): container finished" podID="a10fdfa8-bc02-4e48-8673-9fd121d1222d" containerID="cfa68f01f0249a7c7e1eacc7bedab4197761dcc9e04834d8dfad5222b19a53c4" exitCode=0
Nov 01 09:49:11.005648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:11.005266    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x" event={"ID":"a10fdfa8-bc02-4e48-8673-9fd121d1222d","Type":"ContainerDied","Data":"cfa68f01f0249a7c7e1eacc7bedab4197761dcc9e04834d8dfad5222b19a53c4"}
Nov 01 09:49:11.325677 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:49:11.325626    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:49:12.166879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.166854    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:12.264265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.264179    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca\") pod \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") "
Nov 01 09:49:12.264439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.264353    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh\" (UniqueName: \"kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh\") pod \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") "
Nov 01 09:49:12.264439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.264433    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca\") pod \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\" (UID: \"a10fdfa8-bc02-4e48-8673-9fd121d1222d\") "
Nov 01 09:49:12.264596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.264525    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca" pod "a10fdfa8-bc02-4e48-8673-9fd121d1222d" (UID: "a10fdfa8-bc02-4e48-8673-9fd121d1222d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:49:12.264992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.264970    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:12.266524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.266502    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh" pod "a10fdfa8-bc02-4e48-8673-9fd121d1222d" (UID: "a10fdfa8-bc02-4e48-8673-9fd121d1222d"). InnerVolumeSpecName "kube-api-access-fq4lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:49:12.266524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.266514    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca" pod "a10fdfa8-bc02-4e48-8673-9fd121d1222d" (UID: "a10fdfa8-bc02-4e48-8673-9fd121d1222d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:49:12.366429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.366394    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10fdfa8-bc02-4e48-8673-9fd121d1222d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:12.366429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:12.366423    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4lh\" (UniqueName: \"kubernetes.io/projected/a10fdfa8-bc02-4e48-8673-9fd121d1222d-kube-api-access-fq4lh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:13.019035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:13.019000    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x" event={"ID":"a10fdfa8-bc02-4e48-8673-9fd121d1222d","Type":"ContainerDied","Data":"0dbc71ec12ae4ec7df50d94affd0b82b1a6ddc0922da1923b8b620df7fd66ab5"}
Nov 01 09:49:13.019035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:13.019022    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"
Nov 01 09:49:13.019255 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:13.019054    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dbc71ec12ae4ec7df50d94affd0b82b1a6ddc0922da1923b8b620df7fd66ab5"
Nov 01 09:49:13.190864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:13.190781    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:14.262344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:14.262311    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:14.973739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:14.973708    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:17.999489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:17.999459    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:18.141368 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:18.141326    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:19.170796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:19.170720    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:19.611465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:19.611433    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:19.986900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:19.986869    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:20.142144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:20.142114    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:21.074211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:21.074178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:21.216138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:21.216106    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:21.504042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:21.504003    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:25.071469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:25.071404    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:25.291542 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:25.291505    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:26.458352 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:26.458313    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:27.323801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:27.323757    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:27.477063 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:27.477036    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:27.618906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:27.618829    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:27.921042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:27.920941    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:29.909223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:29.909183    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:33.901685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:33.901648    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:34.045119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:34.045087    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:34.192948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:34.192845    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:35.878211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:35.878178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:39.012245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.011636    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:49:39.318735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.318615    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:49:39.319385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.319347    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Nov 01 09:49:39.325373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.325344    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:49:39.427252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.427193    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:49:39.751252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.751201    4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:49:39.849771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.849723    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 09:49:39.948554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:39.940655    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 01 09:49:40.015615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.014618    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 09:49:40.015615 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.015205    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:49:40.328649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.325627    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:49:40.328649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.326236    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:49:40.397266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.397230    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:49:40.443159 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.443129    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:49:40.870940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:40.870905    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:41.322555 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:49:41.322489    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:49:42.931122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:42.931087    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:43.176419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:43.176385    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:44.484947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:44.484913    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:44.964871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:44.964786    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"]
Nov 01 09:49:44.965061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:44.964977    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy" containerID="cri-o://20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741" gracePeriod=30
Nov 01 09:49:44.981775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:44.981750    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:45.973605 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:45.973542    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:47.932677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:47.932641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:48.913705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.913668    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"]
Nov 01 09:49:48.914253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.914239    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a10fdfa8-bc02-4e48-8673-9fd121d1222d" containerName="revoke-keys"
Nov 01 09:49:48.914299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.914254    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10fdfa8-bc02-4e48-8673-9fd121d1222d" containerName="revoke-keys"
Nov 01 09:49:48.914359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.914350    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10fdfa8-bc02-4e48-8673-9fd121d1222d" containerName="revoke-keys"
Nov 01 09:49:48.917816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.917793    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.950315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.950278    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.950884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.950341    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.950884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.950454    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.950884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.950490    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.950884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.950522    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.950884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.950683    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsz7\" (UniqueName: \"kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:48.968393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.968357    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"openshift-service-ca.crt\""
Nov 01 09:49:48.969270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.969233    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"default-dockercfg-6tkpr\""
Nov 01 09:49:48.969270 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.969241    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"kube-root-ca.crt\""
Nov 01 09:49:48.969432 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.969247    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"rate-test-model-a87a1d-kserve-self-signed-certs\""
Nov 01 09:49:48.973064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:48.973040    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"]
Nov 01 09:49:49.002227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.002167    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:49.052202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.052159    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.052391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.052223    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.052571 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.052515    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.052676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.052619    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsz7\" (UniqueName: \"kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.052676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.052652    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.052798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.052721    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.053423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.053398    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.053888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.053855    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.054171 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.054146    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.059923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.059893    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.066516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.066450    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.110273 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.110219    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsz7\" (UniqueName: \"kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7\") pod \"rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.230462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.230427    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:49.606113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:49.606090    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"]
Nov 01 09:49:49.607486 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:49:49.607453    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod826f3367_0030_46d6_8a81_d94de3680de8.slice/crio-73965629d95ddc8fa614e8412c81eec8aa48788b10e105bcd544c98be6dcd03f WatchSource:0}: Error finding container 73965629d95ddc8fa614e8412c81eec8aa48788b10e105bcd544c98be6dcd03f: Status 404 returned error can't find the container with id 73965629d95ddc8fa614e8412c81eec8aa48788b10e105bcd544c98be6dcd03f
Nov 01 09:49:50.242908 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.242884    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:49:50.274512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274477    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.274731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274581    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.274731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274608    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.274731 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274707    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.275056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274737    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.275056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274920    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.275056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274945    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:49:50.275056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.274961    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.275056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275005    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.275359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275003    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:49:50.275359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275018    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:49:50.275359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275111    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:49:50.275359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275162    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket\") pod \"0ada1f59-3446-416f-900c-4e7a192b4399\" (UID: \"0ada1f59-3446-416f-900c-4e7a192b4399\") "
Nov 01 09:49:50.275609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275467    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:49:50.275715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275680    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.275715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275704    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.275869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275720    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ada1f59-3446-416f-900c-4e7a192b4399-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.275869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275734    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.275869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.275746    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.277187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.277163    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:49:50.277304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.277273    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "kube-api-access-wrlh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:49:50.277390 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.277369    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:49:50.277444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.277388    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo" pod "0ada1f59-3446-416f-900c-4e7a192b4399" (UID: "0ada1f59-3446-416f-900c-4e7a192b4399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:49:50.377428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.377396    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ada1f59-3446-416f-900c-4e7a192b4399-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.377428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.377427    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrlh5\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-kube-api-access-wrlh5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.377428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.377436    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ada1f59-3446-416f-900c-4e7a192b4399-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.377681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.377445    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ada1f59-3446-416f-900c-4e7a192b4399-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:50.494921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.494819    4314 generic.go:356] "Generic (PLEG): container finished" podID="0ada1f59-3446-416f-900c-4e7a192b4399" containerID="20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741" exitCode=0
Nov 01 09:49:50.494921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.494851    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" event={"ID":"0ada1f59-3446-416f-900c-4e7a192b4399","Type":"ContainerDied","Data":"20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741"}
Nov 01 09:49:50.494921 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.494884    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"
Nov 01 09:49:50.495184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.494933    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx" event={"ID":"0ada1f59-3446-416f-900c-4e7a192b4399","Type":"ContainerDied","Data":"88f88006535833a67bcc9f26acfbf0e380283b45078c64d2f826d4bdfc650b69"}
Nov 01 09:49:50.495184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.494956    4314 scope.go:122] "RemoveContainer" containerID="20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741"
Nov 01 09:49:50.496851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.496818    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" event={"ID":"826f3367-0030-46d6-8a81-d94de3680de8","Type":"ContainerStarted","Data":"8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613"}
Nov 01 09:49:50.496953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.496853    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" event={"ID":"826f3367-0030-46d6-8a81-d94de3680de8","Type":"ContainerStarted","Data":"73965629d95ddc8fa614e8412c81eec8aa48788b10e105bcd544c98be6dcd03f"}
Nov 01 09:49:50.506528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.506511    4314 scope.go:122] "RemoveContainer" containerID="abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"
Nov 01 09:49:50.515660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.515641    4314 scope.go:122] "RemoveContainer" containerID="20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741"
Nov 01 09:49:50.515939 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:49:50.515915    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741\": container with ID starting with 20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741 not found: ID does not exist" containerID="20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741"
Nov 01 09:49:50.515999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.515950    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741"} err="failed to get container status \"20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741\": rpc error: code = NotFound desc = could not find container \"20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741\": container with ID starting with 20af91cd65637d21634bac8640a847085348396492d1daaa5e10b7a102b17741 not found: ID does not exist"
Nov 01 09:49:50.515999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.515968    4314 scope.go:122] "RemoveContainer" containerID="abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"
Nov 01 09:49:50.516256 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:49:50.516235    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589\": container with ID starting with abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589 not found: ID does not exist" containerID="abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"
Nov 01 09:49:50.516331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.516259    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589"} err="failed to get container status \"abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589\": rpc error: code = NotFound desc = could not find container \"abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589\": container with ID starting with abb5dbb53165c0add4a025465074b9791f9a60eb65315bd9f9a1f1322fb65589 not found: ID does not exist"
Nov 01 09:49:50.657317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.657250    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" podStartSLOduration=2.657230722 podStartE2EDuration="2.657230722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:49:50.652050072 +0000 UTC m=+4663.846444628" watchObservedRunningTime="2026-11-01 09:49:50.657230722 +0000 UTC m=+4663.851625277"
Nov 01 09:49:50.757318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.757233    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"]
Nov 01 09:49:50.761160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.761131    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3710de-openshift-default-75d4dbc46c-jszvx"]
Nov 01 09:49:50.931790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.931756    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" path="/var/lib/kubelet/pods/0ada1f59-3446-416f-900c-4e7a192b4399/volumes"
Nov 01 09:49:50.978198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:50.978158    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:51.230845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:51.230818    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:55.744765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:55.744734    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:49:59.230663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:59.230628    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:59.243220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:59.243194    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:49:59.567318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:49:59.567238    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:50:00.568638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:00.568594    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:02.096575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:02.096525    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"]
Nov 01 09:50:02.099496 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:02.099474    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f01411f-nqvqs"]
Nov 01 09:50:02.928802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:02.928769    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa01d803-629f-4bd1-a5e8-afea9c1f417c" path="/var/lib/kubelet/pods/fa01d803-629f-4bd1-a5e8-afea9c1f417c/volumes"
Nov 01 09:50:05.433508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:05.433475    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:10.301020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:10.300985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:10.836693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:10.836662    4314 scope.go:122] "RemoveContainer" containerID="79cc6882bd41c925056320b3dcae203de6812361e3bad6db3e0da9ca0059a65f"
Nov 01 09:50:11.323739 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:50:11.323683    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:50:15.216237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:15.216205    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:20.084798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:20.084764    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:22.325511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:22.325463    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:22.475146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:22.475115    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:24.005581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:24.005523    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:24.160926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:24.160894    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:24.897634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:24.897601    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:26.818527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:26.818483    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:29.690823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:29.690780    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:34.582388 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:34.582338    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:34.876961 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:34.876875    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:38.693307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:38.693256    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:38.859596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:38.859563    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:39.491274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:39.491236    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:39.642962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:39.642926    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:41.323494 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:50:41.323447    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:50:44.380442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:44.380401    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:48.836948 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.836862    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"]
Nov 01 09:50:48.837804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.837782    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy"
Nov 01 09:50:48.837891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.837806    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy"
Nov 01 09:50:48.837891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.837822    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy"
Nov 01 09:50:48.837891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.837831    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy"
Nov 01 09:50:48.838026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.837970    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy"
Nov 01 09:50:48.838026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.837984    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ada1f59-3446-416f-900c-4e7a192b4399" containerName="istio-proxy"
Nov 01 09:50:48.842218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.842195    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.893321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.893288    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"]
Nov 01 09:50:48.907815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.907786    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqk6b\" (UniqueName: \"kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.907968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.907843    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.908048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.908021    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.908088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.908075    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.908155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.908138    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.908197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.908178    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:48.935316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.935286    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"rate-test-model-65aeef-kserve-self-signed-certs\""
Nov 01 09:50:48.936527 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.936508    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"openshift-service-ca.crt\""
Nov 01 09:50:48.936619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.936582    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"kube-root-ca.crt\""
Nov 01 09:50:48.936619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.936600    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"default-dockercfg-qvwzz\""
Nov 01 09:50:48.939251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:48.939227    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:49.009757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.009722    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.009955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.009764    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.009955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.009874    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.010087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.009971    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqk6b\" (UniqueName: \"kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.010087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.010020    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.010205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.010122    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.010205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.010165    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.010326 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.010262    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.010384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.010362    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.012156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.012130    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.012416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.012399    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.067000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.066977    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqk6b\" (UniqueName: \"kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b\") pod \"rate-test-model-65aeef-kserve-7b976df745-mt7nl\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.155192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.155119    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:49.190430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.190391    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:49.532009 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.531975    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"]
Nov 01 09:50:49.533328 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:50:49.533300    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod554f1092_3305_461f_9ed5_b29740798074.slice/crio-fdf75ca589b87f3e33919d51ba8aaf4ab0f2132c4d9f1f4dd01a0e3e54319000 WatchSource:0}: Error finding container fdf75ca589b87f3e33919d51ba8aaf4ab0f2132c4d9f1f4dd01a0e3e54319000: Status 404 returned error can't find the container with id fdf75ca589b87f3e33919d51ba8aaf4ab0f2132c4d9f1f4dd01a0e3e54319000
Nov 01 09:50:49.893149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.893041    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" event={"ID":"554f1092-3305-461f-9ed5-b29740798074","Type":"ContainerStarted","Data":"46b8b734f476a1a852f1aff36be161105bb740ed26682f72a7d6c8c26b32b51e"}
Nov 01 09:50:49.893149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:49.893089    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" event={"ID":"554f1092-3305-461f-9ed5-b29740798074","Type":"ContainerStarted","Data":"fdf75ca589b87f3e33919d51ba8aaf4ab0f2132c4d9f1f4dd01a0e3e54319000"}
Nov 01 09:50:50.004928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.004875    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" podStartSLOduration=2.004862195 podStartE2EDuration="2.004862195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:50:50.001700271 +0000 UTC m=+4723.196094826" watchObservedRunningTime="2026-11-01 09:50:50.004862195 +0000 UTC m=+4723.199256750"
Nov 01 09:50:50.094823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.094791    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"]
Nov 01 09:50:50.099050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.099025    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.149070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.148635    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:50:50.149526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.149047    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:50:50.149526 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.149472    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:50:50.149828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.149480    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:50:50.152819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.152794    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"]
Nov 01 09:50:50.155652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.155632    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:50.229450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.229417    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.229450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.229452    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdw4\" (UniqueName: \"kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.229698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.229570    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.330660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.330619    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.330839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.330670    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdw4\" (UniqueName: \"kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.330839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.330733    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.331444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.331415    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.333249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.333224    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.387215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.387190    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdw4\" (UniqueName: \"kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4\") pod \"maas-api-revoke-sub-0076559e-5bn55\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.413462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.413412    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:50.787332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.787300    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"]
Nov 01 09:50:50.788881 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:50:50.788849    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abe6ae1_7752_4075_bdfd_16ac2dcc4e01.slice/crio-3a10bc2e841811bc99b1975647f62f7ab43dce402bc8bcf2859c891529a4d323 WatchSource:0}: Error finding container 3a10bc2e841811bc99b1975647f62f7ab43dce402bc8bcf2859c891529a4d323: Status 404 returned error can't find the container with id 3a10bc2e841811bc99b1975647f62f7ab43dce402bc8bcf2859c891529a4d323
Nov 01 09:50:50.900609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:50.900578    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55" event={"ID":"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01","Type":"ContainerStarted","Data":"3a10bc2e841811bc99b1975647f62f7ab43dce402bc8bcf2859c891529a4d323"}
Nov 01 09:50:51.908343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:51.908245    4314 generic.go:356] "Generic (PLEG): container finished" podID="1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" containerID="b2e041864295536ff5c3dd0795afd0df2977f78492816440cc05bf2311b2e267" exitCode=0
Nov 01 09:50:51.908343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:51.908284    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55" event={"ID":"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01","Type":"ContainerDied","Data":"b2e041864295536ff5c3dd0795afd0df2977f78492816440cc05bf2311b2e267"}
Nov 01 09:50:53.072116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.072092    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:53.168671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.168597    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca\") pod \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") "
Nov 01 09:50:53.168816 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.168703    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca\") pod \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") "
Nov 01 09:50:53.168862 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.168832    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4\" (UniqueName: \"kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4\") pod \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\" (UID: \"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01\") "
Nov 01 09:50:53.169044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.169022    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca" pod "1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" (UID: "1abe6ae1-7752-4075-bdfd-16ac2dcc4e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:50:53.169235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.169218    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:50:53.170664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.170624    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca" pod "1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" (UID: "1abe6ae1-7752-4075-bdfd-16ac2dcc4e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:50:53.170851 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.170832    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4" pod "1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" (UID: "1abe6ae1-7752-4075-bdfd-16ac2dcc4e01"). InnerVolumeSpecName "kube-api-access-pmdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:50:53.270705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.270675    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdw4\" (UniqueName: \"kubernetes.io/projected/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-kube-api-access-pmdw4\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:50:53.270705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.270702    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:50:53.923765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.923723    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55" event={"ID":"1abe6ae1-7752-4075-bdfd-16ac2dcc4e01","Type":"ContainerDied","Data":"3a10bc2e841811bc99b1975647f62f7ab43dce402bc8bcf2859c891529a4d323"}
Nov 01 09:50:53.923765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.923750    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"
Nov 01 09:50:53.924002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:53.923779    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a10bc2e841811bc99b1975647f62f7ab43dce402bc8bcf2859c891529a4d323"
Nov 01 09:50:54.013062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:54.013021    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:58.888839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:58.888802    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:50:59.155926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:59.155842    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:59.168893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:59.168863    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:50:59.976804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:50:59.976772    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:51:00.905293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:00.905252    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:03.675602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:03.675565    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:05.869083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:05.869050    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:07.009577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:07.009520    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:08.610227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:08.610186    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:10.865520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:10.865488    4314 scope.go:122] "RemoveContainer" containerID="805290ea5354b09fe156ee6729c626f33d5f91471fc6c5cadf4deac442c93756"
Nov 01 09:51:11.325698 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:51:11.325650    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:51:13.048450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:13.048414    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:13.517574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:13.517517    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:18.417744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:18.417693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:23.168041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:23.167990    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:27.160813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:27.160755    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:27.931901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:27.931865    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:32.790774 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:32.790737    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:36.297562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.297492    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"]
Nov 01 09:51:36.298145 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.298124    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" containerName="revoke-keys"
Nov 01 09:51:36.298193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.298146    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" containerName="revoke-keys"
Nov 01 09:51:36.298278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.298269    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" containerName="revoke-keys"
Nov 01 09:51:36.302426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.302399    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.351885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.351852    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-65aeef-openshift-default-dockercfg-5fxbb\""
Nov 01 09:51:36.401711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.401673    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"]
Nov 01 09:51:36.410724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.410684    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.410935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.410816    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.410927    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411019 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.410980    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.411068    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.411103    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfcj\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.411164    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.411250    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.411319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.411309    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.512806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.512772    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.512831    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.512888    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513092    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513164    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513250    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513296    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513365 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513322    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513399    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513436    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfcj\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513570    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513595    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513720    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.513820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.513764    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.515620 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.515595    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.515730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.515718    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.571091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.571023    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.571237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.571124    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfcj\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:36.616692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:36.616656    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:37.038644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.038611    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"]
Nov 01 09:51:37.040458 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:51:37.040421    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21560010_c52f_468f_b441_9a3f8ac3e2fd.slice/crio-76c9a31825f999d229b594133cdaf4835c5f59768d5919a06ac6d5d7d90aadb8 WatchSource:0}: Error finding container 76c9a31825f999d229b594133cdaf4835c5f59768d5919a06ac6d5d7d90aadb8: Status 404 returned error can't find the container with id 76c9a31825f999d229b594133cdaf4835c5f59768d5919a06ac6d5d7d90aadb8
Nov 01 09:51:37.043165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.043132    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:51:37.043256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.043205    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:51:37.043256 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.043237    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:51:37.208109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.208071    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" event={"ID":"21560010-c52f-468f-b441-9a3f8ac3e2fd","Type":"ContainerStarted","Data":"fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a"}
Nov 01 09:51:37.208497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.208135    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" event={"ID":"21560010-c52f-468f-b441-9a3f8ac3e2fd","Type":"ContainerStarted","Data":"76c9a31825f999d229b594133cdaf4835c5f59768d5919a06ac6d5d7d90aadb8"}
Nov 01 09:51:37.323138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.323010    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podStartSLOduration=1.322989563 podStartE2EDuration="1.322989563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:51:37.320189084 +0000 UTC m=+4770.514583640" watchObservedRunningTime="2026-11-01 09:51:37.322989563 +0000 UTC m=+4770.517384120"
Nov 01 09:51:37.617139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.617056    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:37.618148 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:37.618126    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:38.619594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:38.619554    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Nov 01 09:51:38.620026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:38.619611    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Nov 01 09:51:39.617012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:39.616977    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Nov 01 09:51:39.617211 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:39.617038    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Nov 01 09:51:40.617778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:40.617741    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Nov 01 09:51:40.618205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:40.617794    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Nov 01 09:51:41.324854 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:51:41.324810    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:51:41.617879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:41.617791    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Nov 01 09:51:41.617879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:41.617848    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Nov 01 09:51:42.506387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:42.506353    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:42.617918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:42.617888    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Nov 01 09:51:42.618353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:42.617937    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Nov 01 09:51:43.678878 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:43.678845    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:43.679631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:43.679292    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:43.680152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:43.680133    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:51:47.369631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:47.369592    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:52.390596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:52.390512    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:51:54.832557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:54.831609    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:51:57.239520 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:51:57.239455    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:02.052059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:02.052015    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:06.888578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:06.888544    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:10.890641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:10.890601    4314 scope.go:122] "RemoveContainer" containerID="2d4aea3250eab3d895af8ecb01c51956ad6cdf7b7769fabf5346ef7a8005d6f8"
Nov 01 09:52:10.900988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:10.900964    4314 scope.go:122] "RemoveContainer" containerID="65d5a2858c39e7bd74dcc57e5d59d8cd0f5fea00c1844b11d7c4a4d3cf9463cb"
Nov 01 09:52:11.327893 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:52:11.327844    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:52:11.721434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:11.721348    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:16.630726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:16.630694    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:21.501986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:21.501914    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:26.513279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:26.513237    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:27.607630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:27.607602    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:31.632740 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:31.632709    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:31.862254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:31.862221    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:32.139894 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.139854    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"]
Nov 01 09:52:32.144068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.144043    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.205667 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.205634    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.206004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.205981    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.206193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.206176    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.206334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.206321    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.206488 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.206475    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.206681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.206660    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.206955 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.206930    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6rq\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.207218 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.207199    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.207497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.207478    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.265563 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.234794    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0401bd23-openshift-default-dockercfg-gvrzh\""
Nov 01 09:52:32.267888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.251417    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"]
Nov 01 09:52:32.293299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.293269    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:32.366577 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366524    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.366795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366623    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.366795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366653    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.366795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366690    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.366795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366712    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.366795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366740    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.366795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366776    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366842    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6rq\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.366933    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367176 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.367123    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.367205    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.367281    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.367345    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.367806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.367782    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.368992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.368976    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.369375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.369358    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.428085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.428008    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.429745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.429721    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6rq\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq\") pod \"e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") " pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.458915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.458885    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:32.548747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.548714    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:32.892121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.892094    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"]
Nov 01 09:52:32.893207 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:52:32.893180    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bf61f8_2793_493b_8ef8_5da66598ab2c.slice/crio-07ea0dacbf5d4885f3eb98e4726a1207f5b3e13fb90c43e35a5714cf1b845927 WatchSource:0}: Error finding container 07ea0dacbf5d4885f3eb98e4726a1207f5b3e13fb90c43e35a5714cf1b845927: Status 404 returned error can't find the container with id 07ea0dacbf5d4885f3eb98e4726a1207f5b3e13fb90c43e35a5714cf1b845927
Nov 01 09:52:32.895596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.895564    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:52:32.895655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.895629    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:52:32.895692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:32.895661    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:52:33.644276 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:33.644237    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" event={"ID":"f9bf61f8-2793-493b-8ef8-5da66598ab2c","Type":"ContainerStarted","Data":"5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"}
Nov 01 09:52:33.644466 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:33.644306    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" event={"ID":"f9bf61f8-2793-493b-8ef8-5da66598ab2c","Type":"ContainerStarted","Data":"07ea0dacbf5d4885f3eb98e4726a1207f5b3e13fb90c43e35a5714cf1b845927"}
Nov 01 09:52:34.459892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.459861    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:34.465631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.465607    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:34.592338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.592308    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:34.650938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.650910    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:34.651752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.651733    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:52:34.664319 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.664262    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podStartSLOduration=2.664249141 podStartE2EDuration="2.664249141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:52:33.809060694 +0000 UTC m=+4827.003455260" watchObservedRunningTime="2026-11-01 09:52:34.664249141 +0000 UTC m=+4827.858643695"
Nov 01 09:52:34.896300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:34.896262    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:36.674225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:36.674194    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:36.882799 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:36.882760    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:37.121444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:37.121411    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:37.341360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:37.341325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:40.252213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.252182    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:40.444451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.444412    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:40.827133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.826425    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"]
Nov 01 09:52:40.832047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.832005    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:40.894118 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.894088    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0401bd23\""
Nov 01 09:52:40.894272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.894138    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0401bd23-dockercfg-bdzrw\""
Nov 01 09:52:40.904852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.904824    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume\") pod \"payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") " pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:40.905003 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.904882    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztml\" (UniqueName: \"kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml\") pod \"payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") " pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:40.946031 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:40.946002    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"]
Nov 01 09:52:41.006594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.006518    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume\") pod \"payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") " pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:41.006805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.006647    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztml\" (UniqueName: \"kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml\") pod \"payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") " pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:41.007321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.007302    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume\") pod \"payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") " pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:41.069754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.069722    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztml\" (UniqueName: \"kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml\") pod \"payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") " pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:41.153085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.152999    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:41.324525 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:52:41.324491    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:52:41.534885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.534853    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"]
Nov 01 09:52:41.535453 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:52:41.535424    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb071d463_a17c_4535_9a8e_cde5581d4275.slice/crio-ee7a07a9e0a9efaa3aa4b5e0a0171905dc773f71364a7288c59e86191d2730e5 WatchSource:0}: Error finding container ee7a07a9e0a9efaa3aa4b5e0a0171905dc773f71364a7288c59e86191d2730e5: Status 404 returned error can't find the container with id ee7a07a9e0a9efaa3aa4b5e0a0171905dc773f71364a7288c59e86191d2730e5
Nov 01 09:52:41.699664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.699629    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6" event={"ID":"b071d463-a17c-4535-9a8e-cde5581d4275","Type":"ContainerStarted","Data":"f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1"}
Nov 01 09:52:41.699869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.699678    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6" event={"ID":"b071d463-a17c-4535-9a8e-cde5581d4275","Type":"ContainerStarted","Data":"ee7a07a9e0a9efaa3aa4b5e0a0171905dc773f71364a7288c59e86191d2730e5"}
Nov 01 09:52:41.699869 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.699845    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:41.824200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:41.824096    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6" podStartSLOduration=1.824080468 podStartE2EDuration="1.824080468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:52:41.820893739 +0000 UTC m=+4835.015288295" watchObservedRunningTime="2026-11-01 09:52:41.824080468 +0000 UTC m=+4835.018475077"
Nov 01 09:52:42.439686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:42.439649    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:43.392959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:43.392922    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:44.493340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:44.493309    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:44.684509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:44.684472    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:45.694913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:45.694880    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:47.791473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:47.791437    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:51.360230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:51.360200    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:52.606205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:52.606170    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:52:52.710170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:52.710134    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:52:57.396239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:52:57.396206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:02.297305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:02.297260    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:07.093994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:07.093963    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:10.933308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:10.933276    4314 scope.go:122] "RemoveContainer" containerID="6dd0d39def1c51b0c904d41d22146caec7e12c570dee3331c35ad852c6b73997"
Nov 01 09:53:10.943575 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:10.943556    4314 scope.go:122] "RemoveContainer" containerID="041a16242ff36a13cee52df9244bd71247dbcb44aa27f74ab72ee891b034a135"
Nov 01 09:53:11.326632 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:53:11.326595    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:53:12.007836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:12.007804    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:13.384648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:13.384608    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:16.867761 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:16.867728    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:17.033953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:17.033917    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:17.339557 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:17.339491    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:22.227146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:22.227112    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:26.500664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:26.500631    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:27.762047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:27.762007    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:27.941489 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:27.941451    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:28.229516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:28.229428    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:28.413429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:28.413397    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:29.125372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:29.125339    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:29.506911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:29.506883    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:30.002629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:30.002589    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"]
Nov 01 09:53:30.002925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:30.002895    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" containerID="cri-o://f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e" gracePeriod=30
Nov 01 09:53:30.037479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:30.037452    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:30.847849 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:30.847814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:32.838217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:32.838182    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:53:32.838656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:32.838249    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:53:33.854649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:33.854607    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:35.270552 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.270473    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:53:35.398736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.398706    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.398941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.398764    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.398941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.398825    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.398941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.398875    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.398941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.398926    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.399182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.398973    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.399182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399048    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:53:35.399182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399060    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.399182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399157    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:53:35.399413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399240    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.399413 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399330    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token\") pod \"d73eaa51-7bee-44b5-b178-c774c7984717\" (UID: \"d73eaa51-7bee-44b5-b178-c774c7984717\") "
Nov 01 09:53:35.399706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399675    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:53:35.400249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.399834    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:53:35.400249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.400103    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.400249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.400125    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eaa51-7bee-44b5-b178-c774c7984717-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.400249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.400140    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.400249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.400153    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.400860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.400831    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:53:35.401694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.401670    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:53:35.401694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.401679    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "kube-api-access-ds26h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:53:35.402004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.401976    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:53:35.402109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.402075    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token" pod "d73eaa51-7bee-44b5-b178-c774c7984717" (UID: "d73eaa51-7bee-44b5-b178-c774c7984717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:53:35.501562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.501504    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.501562 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.501565    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.501765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.501580    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eaa51-7bee-44b5-b178-c774c7984717-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.501765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.501593    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eaa51-7bee-44b5-b178-c774c7984717-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:35.501765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:35.501605    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds26h\" (UniqueName: \"kubernetes.io/projected/d73eaa51-7bee-44b5-b178-c774c7984717-kube-api-access-ds26h\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:36.070170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.070141    4314 generic.go:356] "Generic (PLEG): container finished" podID="d73eaa51-7bee-44b5-b178-c774c7984717" containerID="f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e" exitCode=0
Nov 01 09:53:36.070362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.070196    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"
Nov 01 09:53:36.070362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.070233    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" event={"ID":"d73eaa51-7bee-44b5-b178-c774c7984717","Type":"ContainerDied","Data":"f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e"}
Nov 01 09:53:36.070362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.070288    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj" event={"ID":"d73eaa51-7bee-44b5-b178-c774c7984717","Type":"ContainerDied","Data":"c8b6f8055726748cd8c3daed863c55138946160033bc50ae428a7e061744da69"}
Nov 01 09:53:36.070362 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.070312    4314 scope.go:122] "RemoveContainer" containerID="f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e"
Nov 01 09:53:36.081343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.081326    4314 scope.go:122] "RemoveContainer" containerID="32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"
Nov 01 09:53:36.090794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.090772    4314 scope.go:122] "RemoveContainer" containerID="f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e"
Nov 01 09:53:36.091092 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:53:36.091067    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e\": container with ID starting with f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e not found: ID does not exist" containerID="f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e"
Nov 01 09:53:36.091184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.091099    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e"} err="failed to get container status \"f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e\": rpc error: code = NotFound desc = could not find container \"f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e\": container with ID starting with f96e99a88211d7ea5c88e16948c9167e8f1e6521dca64db6a6c78a039827b52e not found: ID does not exist"
Nov 01 09:53:36.091184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.091119    4314 scope.go:122] "RemoveContainer" containerID="32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"
Nov 01 09:53:36.091473 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:53:36.091454    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e\": container with ID starting with 32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e not found: ID does not exist" containerID="32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"
Nov 01 09:53:36.091543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.091481    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e"} err="failed to get container status \"32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e\": rpc error: code = NotFound desc = could not find container \"32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e\": container with ID starting with 32de6b7b48c93123559649487a5c47f30b38560a8df27a2be3bcd0b93851b39e not found: ID does not exist"
Nov 01 09:53:36.232931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.232898    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"]
Nov 01 09:53:36.239122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.239095    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-035625-openshift-default-bd5595c9-98skj"]
Nov 01 09:53:36.932361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:36.932326    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" path="/var/lib/kubelet/pods/d73eaa51-7bee-44b5-b178-c774c7984717/volumes"
Nov 01 09:53:38.271516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:38.271476    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:38.451939 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:38.451906    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"]
Nov 01 09:53:38.776167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:38.776136    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:39.025785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:39.025747    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:39.229905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:39.229822    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:40.212707 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:40.212669    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:40.447442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:40.447405    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:40.636709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:40.636677    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:40.818157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:40.818126    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:41.324611 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:53:41.324573    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:53:43.452811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:43.452748    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerName="payload-pre-processing" containerID="cri-o://7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d" gracePeriod=25
Nov 01 09:53:44.579699 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:44.579644    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:53:45.586874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:45.586829    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:46.717631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:46.717595    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:48.969461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:48.969375    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:50.657284 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:50.657250    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:53.476910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:53.476869    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:53.660335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:53.660301    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:54.550085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:54.550041    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:53:55.096246 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:55.096205    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:55.506170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:55.506139    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"]
Nov 01 09:53:55.506385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:55.506349    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" containerID="cri-o://421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd" gracePeriod=30
Nov 01 09:53:55.556093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:55.556059    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:55.813467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:55.813384    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:53:55.813467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:55.813442    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:53:56.004979 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.004945    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:56.164091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.164011    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:53:56.588201 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.588170    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"]
Nov 01 09:53:56.588798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.588780    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy"
Nov 01 09:53:56.588798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.588799    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy"
Nov 01 09:53:56.588945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.588911    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy"
Nov 01 09:53:56.588945 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.588925    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy"
Nov 01 09:53:56.592350 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.592329    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.643779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.643753    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:53:56.643926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.643794    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:53:56.644749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.644727    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:53:56.644887 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.644791    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:53:56.647897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.647863    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"]
Nov 01 09:53:56.669779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.669749    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.669971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.669950    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.670041 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.669997    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfxx\" (UniqueName: \"kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.770965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.770930    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.770965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.770970    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfxx\" (UniqueName: \"kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.771185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.771070    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.771766 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.771745    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.773306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.773286    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.828787 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.828757    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfxx\" (UniqueName: \"kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx\") pod \"maas-api-revoke-sub-e7d9e6d9-lctdk\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:56.906555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:56.906459    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:57.279978 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:57.279948    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"]
Nov 01 09:53:57.280860 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:53:57.280831    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ce3542_b285_4cb9_8684_01daae4f0efe.slice/crio-494c715ebf5415effd4f53695d82bcea73e7496feb1382a1a0627548845ee0a8 WatchSource:0}: Error finding container 494c715ebf5415effd4f53695d82bcea73e7496feb1382a1a0627548845ee0a8: Status 404 returned error can't find the container with id 494c715ebf5415effd4f53695d82bcea73e7496feb1382a1a0627548845ee0a8
Nov 01 09:53:58.224047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:58.224009    4314 generic.go:356] "Generic (PLEG): container finished" podID="11ce3542-b285-4cb9-8684-01daae4f0efe" containerID="54918fb145640ad467fcde6431040ac7b44d4e4a8410f50b35b60397d33ac2bc" exitCode=0
Nov 01 09:53:58.224447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:58.224091    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk" event={"ID":"11ce3542-b285-4cb9-8684-01daae4f0efe","Type":"ContainerDied","Data":"54918fb145640ad467fcde6431040ac7b44d4e4a8410f50b35b60397d33ac2bc"}
Nov 01 09:53:58.224447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:58.224168    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk" event={"ID":"11ce3542-b285-4cb9-8684-01daae4f0efe","Type":"ContainerStarted","Data":"494c715ebf5415effd4f53695d82bcea73e7496feb1382a1a0627548845ee0a8"}
Nov 01 09:53:59.386697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.386671    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:53:59.507271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.507177    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca\") pod \"11ce3542-b285-4cb9-8684-01daae4f0efe\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") "
Nov 01 09:53:59.507271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.507233    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca\") pod \"11ce3542-b285-4cb9-8684-01daae4f0efe\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") "
Nov 01 09:53:59.507468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.507389    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx\" (UniqueName: \"kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx\") pod \"11ce3542-b285-4cb9-8684-01daae4f0efe\" (UID: \"11ce3542-b285-4cb9-8684-01daae4f0efe\") "
Nov 01 09:53:59.507673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.507646    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca" pod "11ce3542-b285-4cb9-8684-01daae4f0efe" (UID: "11ce3542-b285-4cb9-8684-01daae4f0efe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:53:59.508014 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.507984    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:59.509561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.509511    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca" pod "11ce3542-b285-4cb9-8684-01daae4f0efe" (UID: "11ce3542-b285-4cb9-8684-01daae4f0efe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:53:59.509653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.509618    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx" pod "11ce3542-b285-4cb9-8684-01daae4f0efe" (UID: "11ce3542-b285-4cb9-8684-01daae4f0efe"). InnerVolumeSpecName "kube-api-access-tqfxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:53:59.608951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.608919    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfxx\" (UniqueName: \"kubernetes.io/projected/11ce3542-b285-4cb9-8684-01daae4f0efe-kube-api-access-tqfxx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:59.608951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:53:59.608949    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce3542-b285-4cb9-8684-01daae4f0efe-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:00.238987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.238951    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk" event={"ID":"11ce3542-b285-4cb9-8684-01daae4f0efe","Type":"ContainerDied","Data":"494c715ebf5415effd4f53695d82bcea73e7496feb1382a1a0627548845ee0a8"}
Nov 01 09:54:00.238987 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.238970    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"
Nov 01 09:54:00.239209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.239004    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494c715ebf5415effd4f53695d82bcea73e7496feb1382a1a0627548845ee0a8"
Nov 01 09:54:00.771643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.771578    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:54:00.926011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.925983    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926040    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926118    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926151    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926182    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926236    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926289    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926335    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926368    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket\") pod \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\" (UID: \"94455c39-0ea2-43f4-861d-97e6bc7bfca3\") "
Nov 01 09:54:00.926934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.926894    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:00.927078 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.927006    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:00.927224 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.927172    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:00.927310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.927259    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:00.927619 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.927595    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:00.929116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.928993    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:00.929609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.929585    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:54:00.929778 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.929750    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "kube-api-access-hr94l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:00.930024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.929998    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token" pod "94455c39-0ea2-43f4-861d-97e6bc7bfca3" (UID: "94455c39-0ea2-43f4-861d-97e6bc7bfca3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:00.984071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:00.984035    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"]
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.028993    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029027    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029042    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr94l\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-kube-api-access-hr94l\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029057    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029071    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029084    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029098    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029112    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94455c39-0ea2-43f4-861d-97e6bc7bfca3-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.029206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.029125    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94455c39-0ea2-43f4-861d-97e6bc7bfca3-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:01.111858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.111823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:01.247216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.247180    4314 generic.go:356] "Generic (PLEG): container finished" podID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerID="421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd" exitCode=0
Nov 01 09:54:01.247427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.247225    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"
Nov 01 09:54:01.247427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.247270    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" event={"ID":"94455c39-0ea2-43f4-861d-97e6bc7bfca3","Type":"ContainerDied","Data":"421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd"}
Nov 01 09:54:01.247427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.247331    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw" event={"ID":"94455c39-0ea2-43f4-861d-97e6bc7bfca3","Type":"ContainerDied","Data":"668dee011ead20f6805dda980e2569c257706f8bcfe6d9a5f55799ef7e457043"}
Nov 01 09:54:01.247427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.247360    4314 scope.go:122] "RemoveContainer" containerID="421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd"
Nov 01 09:54:01.258021 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.257993    4314 scope.go:122] "RemoveContainer" containerID="9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"
Nov 01 09:54:01.267366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.267345    4314 scope.go:122] "RemoveContainer" containerID="421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd"
Nov 01 09:54:01.267788 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:54:01.267767    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd\": container with ID starting with 421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd not found: ID does not exist" containerID="421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd"
Nov 01 09:54:01.267859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.267797    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd"} err="failed to get container status \"421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd\": rpc error: code = NotFound desc = could not find container \"421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd\": container with ID starting with 421e2f4c0f293c2c01cfa8511ac1e25357c82ba43f861c6766432ac3dc2ad6fd not found: ID does not exist"
Nov 01 09:54:01.267859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.267817    4314 scope.go:122] "RemoveContainer" containerID="9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"
Nov 01 09:54:01.268067 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:54:01.268050    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c\": container with ID starting with 9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c not found: ID does not exist" containerID="9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"
Nov 01 09:54:01.268113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.268071    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c"} err="failed to get container status \"9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c\": rpc error: code = NotFound desc = could not find container \"9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c\": container with ID starting with 9f66fc2ab09d5822085317da60113e16efd007fa5ca132f5613845520bc6a87c not found: ID does not exist"
Nov 01 09:54:01.270086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.270060    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:01.405130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.405048    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"]
Nov 01 09:54:01.412963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:01.412936    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5b96ac-openshift-default-6d4c67497f-2mfrw"]
Nov 01 09:54:02.080646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:02.080614    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"]
Nov 01 09:54:02.090060 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:02.090034    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b52424-749cr"]
Nov 01 09:54:02.929325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:02.929294    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" path="/var/lib/kubelet/pods/94455c39-0ea2-43f4-861d-97e6bc7bfca3/volumes"
Nov 01 09:54:02.929786 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:02.929771    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74a67a2-000d-4501-a035-462199f9b896" path="/var/lib/kubelet/pods/b74a67a2-000d-4501-a035-462199f9b896/volumes"
Nov 01 09:54:03.734709 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.734684    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:54:03.757001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.756976    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume\") pod \"91720349-4f67-4f40-8a1c-093a6c7b5a94\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") "
Nov 01 09:54:03.757136 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.757072    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm\" (UniqueName: \"kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm\") pod \"91720349-4f67-4f40-8a1c-093a6c7b5a94\" (UID: \"91720349-4f67-4f40-8a1c-093a6c7b5a94\") "
Nov 01 09:54:03.757419 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.757389    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume" pod "91720349-4f67-4f40-8a1c-093a6c7b5a94" (UID: "91720349-4f67-4f40-8a1c-093a6c7b5a94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:03.757874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.757845    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91720349-4f67-4f40-8a1c-093a6c7b5a94-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:03.759678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.759648    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm" pod "91720349-4f67-4f40-8a1c-093a6c7b5a94" (UID: "91720349-4f67-4f40-8a1c-093a6c7b5a94"). InnerVolumeSpecName "kube-api-access-btkxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:03.766304 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.766148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:03.859363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:03.859324    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkxm\" (UniqueName: \"kubernetes.io/projected/91720349-4f67-4f40-8a1c-093a6c7b5a94-kube-api-access-btkxm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:04.270069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.270035    4314 generic.go:356] "Generic (PLEG): container finished" podID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerID="7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d" exitCode=0
Nov 01 09:54:04.270293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.270083    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"
Nov 01 09:54:04.270293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.270106    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" event={"ID":"91720349-4f67-4f40-8a1c-093a6c7b5a94","Type":"ContainerDied","Data":"7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d"}
Nov 01 09:54:04.270293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.270161    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt" event={"ID":"91720349-4f67-4f40-8a1c-093a6c7b5a94","Type":"ContainerDied","Data":"ff63122845ce2d4f4a96df0113b10dc4155f4a5010e3c6a29f87a7994424cec2"}
Nov 01 09:54:04.270293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.270190    4314 scope.go:122] "RemoveContainer" containerID="7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d"
Nov 01 09:54:04.280260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.280235    4314 scope.go:122] "RemoveContainer" containerID="7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d"
Nov 01 09:54:04.280564 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:54:04.280518    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d\": container with ID starting with 7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d not found: ID does not exist" containerID="7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d"
Nov 01 09:54:04.280621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.280573    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d"} err="failed to get container status \"7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d\": rpc error: code = NotFound desc = could not find container \"7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d\": container with ID starting with 7e6c055a36e488a5e0d747764579095e4e6fe110ae6f62964cacb885ab3c613d not found: ID does not exist"
Nov 01 09:54:04.437471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.437434    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"]
Nov 01 09:54:04.483212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.483171    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5b96ac-6768bfc979-qb9xt"]
Nov 01 09:54:04.930070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:04.930038    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" path="/var/lib/kubelet/pods/91720349-4f67-4f40-8a1c-093a6c7b5a94/volumes"
Nov 01 09:54:06.311590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:06.311522    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:07.259814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:07.259777    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:11.272673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:11.272637    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:11.326614 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:54:11.326571    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:54:13.057445 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.057415    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"]
Nov 01 09:54:13.102989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.102955    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2394826-s999x"]
Nov 01 09:54:13.420299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420202    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"]
Nov 01 09:54:13.420915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420881    4314 cpu_manager.go:414] "containerMap: removing container" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy"
Nov 01 09:54:13.420915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420904    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420936    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420941    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73eaa51-7bee-44b5-b178-c774c7984717" containerName="istio-proxy"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420960    4314 cpu_manager.go:414] "containerMap: removing container" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerName="payload-pre-processing"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420965    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerName="payload-pre-processing"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420978    4314 cpu_manager.go:414] "containerMap: removing container" podUID="11ce3542-b285-4cb9-8684-01daae4f0efe" containerName="revoke-keys"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.420982    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ce3542-b285-4cb9-8684-01daae4f0efe" containerName="revoke-keys"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.421086    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="91720349-4f67-4f40-8a1c-093a6c7b5a94" containerName="payload-pre-processing"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.421096    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.421103    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ce3542-b285-4cb9-8684-01daae4f0efe" containerName="revoke-keys"
Nov 01 09:54:13.421137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.421111    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy"
Nov 01 09:54:13.426247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.426224    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.474327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.474291    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.474498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.474354    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.474498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.474458    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntlx5\" (UniqueName: \"kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.481089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.481063    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:54:13.481382 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.481352    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:54:13.482841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.482818    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"]
Nov 01 09:54:13.523852 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.523821    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:54:13.524011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.523873    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:54:13.576052 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.576014    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.576274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.576081    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.576274 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.576117    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntlx5\" (UniqueName: \"kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.576828 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.576804    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.578596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.578579    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.634853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.634818    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntlx5\" (UniqueName: \"kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5\") pod \"maas-api-revoke-sub-f2bf608e-bj2sn\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:13.741497 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:13.741464    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:14.026091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.025987    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:14.115613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.115585    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"]
Nov 01 09:54:14.117710 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:54:14.117678    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080dd537_a63d_4de0_b4a2_2c1406e7005c.slice/crio-f1082ed3b731f881c29d08854e102c2ce9c36a57357d934360b44a40957a297d WatchSource:0}: Error finding container f1082ed3b731f881c29d08854e102c2ce9c36a57357d934360b44a40957a297d: Status 404 returned error can't find the container with id f1082ed3b731f881c29d08854e102c2ce9c36a57357d934360b44a40957a297d
Nov 01 09:54:14.335440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.335405    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"]
Nov 01 09:54:14.339559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.339522    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn" event={"ID":"080dd537-a63d-4de0-b4a2-2c1406e7005c","Type":"ContainerStarted","Data":"f1082ed3b731f881c29d08854e102c2ce9c36a57357d934360b44a40957a297d"}
Nov 01 09:54:14.445798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.445766    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"]
Nov 01 09:54:14.725448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.725362    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:14.894092 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.894057    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:14.929606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:14.929576    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10fdfa8-bc02-4e48-8673-9fd121d1222d" path="/var/lib/kubelet/pods/a10fdfa8-bc02-4e48-8673-9fd121d1222d/volumes"
Nov 01 09:54:15.347183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:15.347106    4314 generic.go:356] "Generic (PLEG): container finished" podID="080dd537-a63d-4de0-b4a2-2c1406e7005c" containerID="a8ba1074635eef0b84f36ef9597fdaf07d9050d4276f0d2a2f8cd2ab7bd6cedb" exitCode=0
Nov 01 09:54:15.347576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:15.347187    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn" event={"ID":"080dd537-a63d-4de0-b4a2-2c1406e7005c","Type":"ContainerDied","Data":"a8ba1074635eef0b84f36ef9597fdaf07d9050d4276f0d2a2f8cd2ab7bd6cedb"}
Nov 01 09:54:16.006591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.006492    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main" containerID="cri-o://21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97" gracePeriod=45
Nov 01 09:54:16.277113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.277084    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:54:16.355809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.355771    4314 generic.go:356] "Generic (PLEG): container finished" podID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerID="21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97" exitCode=0
Nov 01 09:54:16.356372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.355821    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"
Nov 01 09:54:16.356372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.355850    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" event={"ID":"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab","Type":"ContainerDied","Data":"21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97"}
Nov 01 09:54:16.356372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.355900    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp" event={"ID":"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab","Type":"ContainerDied","Data":"1bf34ce80ebe5506e03f1eb0c4cdc4805faa9bb665cb5adb25cacc54d5f97b34"}
Nov 01 09:54:16.356372 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.355919    4314 scope.go:122] "RemoveContainer" containerID="21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97"
Nov 01 09:54:16.366901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.366879    4314 scope.go:122] "RemoveContainer" containerID="21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97"
Nov 01 09:54:16.367226 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:54:16.367198    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97\": container with ID starting with 21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97 not found: ID does not exist" containerID="21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97"
Nov 01 09:54:16.367339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.367235    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97"} err="failed to get container status \"21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97\": rpc error: code = NotFound desc = could not find container \"21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97\": container with ID starting with 21dd63a432e7cf931dd100521115ed53747c46222f00ec6f7d82e7b32b7bfc97 not found: ID does not exist"
Nov 01 09:54:16.423440 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.423402    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm\") pod \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") "
Nov 01 09:54:16.423705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.423481    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir\") pod \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") "
Nov 01 09:54:16.423705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.423524    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs\") pod \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") "
Nov 01 09:54:16.423705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.423600    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home\") pod \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") "
Nov 01 09:54:16.423705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.423642    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache\") pod \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") "
Nov 01 09:54:16.424080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.423735    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4\" (UniqueName: \"kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4\") pod \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\" (UID: \"5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab\") "
Nov 01 09:54:16.424881 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.424690    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home" pod "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" (UID: "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425353 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.425067    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache" pod "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" (UID: "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.425712    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir" pod "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" (UID: "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.428857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.428832    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs" pod "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" (UID: "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:16.429753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.429732    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm" pod "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" (UID: "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.434695 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.434662    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4" pod "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" (UID: "5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab"). InnerVolumeSpecName "kube-api-access-5mdg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:16.497781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.497752    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:16.525461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.525385    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.525461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.525420    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.525461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.525435    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.525461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.525448    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.525461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.525462    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.525757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.525476    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mdg4\" (UniqueName: \"kubernetes.io/projected/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab-kube-api-access-5mdg4\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.626745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.626706    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca\") pod \"080dd537-a63d-4de0-b4a2-2c1406e7005c\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") "
Nov 01 09:54:16.627167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.627114    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca" pod "080dd537-a63d-4de0-b4a2-2c1406e7005c" (UID: "080dd537-a63d-4de0-b4a2-2c1406e7005c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:16.627398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.627378    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5\" (UniqueName: \"kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5\") pod \"080dd537-a63d-4de0-b4a2-2c1406e7005c\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") "
Nov 01 09:54:16.627519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.627501    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca\") pod \"080dd537-a63d-4de0-b4a2-2c1406e7005c\" (UID: \"080dd537-a63d-4de0-b4a2-2c1406e7005c\") "
Nov 01 09:54:16.628469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.628424    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.628469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.631777    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca" pod "080dd537-a63d-4de0-b4a2-2c1406e7005c" (UID: "080dd537-a63d-4de0-b4a2-2c1406e7005c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:16.628469 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.632659    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5" pod "080dd537-a63d-4de0-b4a2-2c1406e7005c" (UID: "080dd537-a63d-4de0-b4a2-2c1406e7005c"). InnerVolumeSpecName "kube-api-access-ntlx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:16.697983 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.697934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:16.730182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.730144    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntlx5\" (UniqueName: \"kubernetes.io/projected/080dd537-a63d-4de0-b4a2-2c1406e7005c-kube-api-access-ntlx5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.730182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.730186    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080dd537-a63d-4de0-b4a2-2c1406e7005c-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.821874 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.821800    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"]
Nov 01 09:54:16.827251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.827221    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-w3-54e01b-kserve-78c6db7969-vcfkp"]
Nov 01 09:54:16.929358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:16.929325    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" path="/var/lib/kubelet/pods/5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab/volumes"
Nov 01 09:54:17.364183 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:17.364153    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"
Nov 01 09:54:17.364642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:17.364145    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn" event={"ID":"080dd537-a63d-4de0-b4a2-2c1406e7005c","Type":"ContainerDied","Data":"f1082ed3b731f881c29d08854e102c2ce9c36a57357d934360b44a40957a297d"}
Nov 01 09:54:17.364642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:17.364314    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1082ed3b731f881c29d08854e102c2ce9c36a57357d934360b44a40957a297d"
Nov 01 09:54:19.336254 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:19.336211    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerName="payload-processing" containerID="cri-o://b3ec34d0a0b4e9e58e53519249c69081318931e0047ec8aca290ee3827989dc4" gracePeriod=25
Nov 01 09:54:19.446896 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:19.446857    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerName="payload-pre-processing" containerID="cri-o://d1190265b9e3ddcc1ac199e9a8fc19abcb13794c15ef751bbaf70e9dc54002f7" gracePeriod=25
Nov 01 09:54:21.743929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:21.743895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:23.138101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:23.138072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:27.044253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.044198    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:27.474113 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.474034    4314 generic.go:356] "Generic (PLEG): container finished" podID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerID="5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7" exitCode=0
Nov 01 09:54:27.474280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.474106    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" event={"ID":"f9bf61f8-2793-493b-8ef8-5da66598ab2c","Type":"ContainerDied","Data":"5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"}
Nov 01 09:54:27.474637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.474596    4314 scope.go:122] "RemoveContainer" containerID="5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"
Nov 01 09:54:27.476189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.476157    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:54:27.476342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.476242    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:54:27.476342 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.476303    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:54:27.727693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.727612    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:54:27.727850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:27.727612    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:54:28.291888 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:28.291851    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:28.484339 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:28.484305    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:28.491182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:28.491153    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" event={"ID":"f9bf61f8-2793-493b-8ef8-5da66598ab2c","Type":"ContainerStarted","Data":"1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e"}
Nov 01 09:54:29.468870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:29.468836    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:54:29.469560 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:29.469511    4314 patch_prober.go:29] interesting pod/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Nov 01 09:54:29.469738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:29.469712    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Nov 01 09:54:29.582030 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:29.581994    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:30.463902 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:30.463864    4314 patch_prober.go:29] interesting pod/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Nov 01 09:54:30.464185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:30.464156    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Nov 01 09:54:31.463865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:31.463826    4314 patch_prober.go:29] interesting pod/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Nov 01 09:54:31.464491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:31.464456    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Nov 01 09:54:32.230581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:32.230549    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:32.471558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:32.463846    4314 patch_prober.go:29] interesting pod/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Nov 01 09:54:32.471558 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:32.463917    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Nov 01 09:54:33.459167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:33.459131    4314 patch_prober.go:29] interesting pod/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Nov 01 09:54:33.459344 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:33.459187    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Nov 01 09:54:34.463710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:34.463680    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:54:34.464197 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:34.464183    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:54:34.464933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:34.464910    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:54:35.608700 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.608668    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"]
Nov 01 09:54:35.609264 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609249    4314 cpu_manager.go:414] "containerMap: removing container" podUID="080dd537-a63d-4de0-b4a2-2c1406e7005c" containerName="revoke-keys"
Nov 01 09:54:35.609310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609265    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080dd537-a63d-4de0-b4a2-2c1406e7005c" containerName="revoke-keys"
Nov 01 09:54:35.609310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609274    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main"
Nov 01 09:54:35.609310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609281    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main"
Nov 01 09:54:35.609310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609298    4314 cpu_manager.go:414] "containerMap: removing container" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy"
Nov 01 09:54:35.609310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609303    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94455c39-0ea2-43f4-861d-97e6bc7bfca3" containerName="istio-proxy"
Nov 01 09:54:35.609484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609391    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="080dd537-a63d-4de0-b4a2-2c1406e7005c" containerName="revoke-keys"
Nov 01 09:54:35.609484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.609399    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5dc9ea-7340-4c0a-8f29-daac7de1c4ab" containerName="main"
Nov 01 09:54:35.611735 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.611718    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.652204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.652173    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55vz\" (UniqueName: \"kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.652345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.652219    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.652444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.652425    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.706127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.706096    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:54:35.706297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.706160    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:54:35.706297 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.706173    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:54:35.706381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.706332    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:54:35.709630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.709608    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"]
Nov 01 09:54:35.753804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.753772    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.753996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.753826    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55vz\" (UniqueName: \"kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.753996 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.753871    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.754451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.754433    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.756373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.756349    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.811724 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.811688    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55vz\" (UniqueName: \"kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz\") pod \"maas-api-revoke-sub-17191c34-tpqmz\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:35.924242 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:35.924162    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:36.293133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:36.293101    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"]
Nov 01 09:54:36.293991 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:54:36.293960    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9805952_12b5_449d_a33c_bc3b089185e3.slice/crio-4fa3810d774011caa466e43212084b2421df15a91ae1125c544a2484299c03f4 WatchSource:0}: Error finding container 4fa3810d774011caa466e43212084b2421df15a91ae1125c544a2484299c03f4: Status 404 returned error can't find the container with id 4fa3810d774011caa466e43212084b2421df15a91ae1125c544a2484299c03f4
Nov 01 09:54:36.548222 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:36.548103    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz" event={"ID":"b9805952-12b5-449d-a33c-bc3b089185e3","Type":"ContainerStarted","Data":"4fa3810d774011caa466e43212084b2421df15a91ae1125c544a2484299c03f4"}
Nov 01 09:54:37.338162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:37.338130    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:37.578122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:37.578083    4314 generic.go:356] "Generic (PLEG): container finished" podID="b9805952-12b5-449d-a33c-bc3b089185e3" containerID="fd959b94402b9c561fab96b389f0f045fb0cec141785d78d84ac79f455c6c42d" exitCode=0
Nov 01 09:54:37.578659 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:37.578268    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz" event={"ID":"b9805952-12b5-449d-a33c-bc3b089185e3","Type":"ContainerDied","Data":"fd959b94402b9c561fab96b389f0f045fb0cec141785d78d84ac79f455c6c42d"}
Nov 01 09:54:37.727692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:37.727657    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:54:37.727861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:37.727657    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:54:38.733451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.733425    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:38.793340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.793309    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca\") pod \"b9805952-12b5-449d-a33c-bc3b089185e3\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") "
Nov 01 09:54:38.793514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.793413    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca\") pod \"b9805952-12b5-449d-a33c-bc3b089185e3\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") "
Nov 01 09:54:38.793603 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.793578    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz\" (UniqueName: \"kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz\") pod \"b9805952-12b5-449d-a33c-bc3b089185e3\" (UID: \"b9805952-12b5-449d-a33c-bc3b089185e3\") "
Nov 01 09:54:38.793803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.793778    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca" pod "b9805952-12b5-449d-a33c-bc3b089185e3" (UID: "b9805952-12b5-449d-a33c-bc3b089185e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:38.794093 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.794066    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:38.795524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.795507    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca" pod "b9805952-12b5-449d-a33c-bc3b089185e3" (UID: "b9805952-12b5-449d-a33c-bc3b089185e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:38.795643 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.795621    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz" pod "b9805952-12b5-449d-a33c-bc3b089185e3" (UID: "b9805952-12b5-449d-a33c-bc3b089185e3"). InnerVolumeSpecName "kube-api-access-f55vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:38.895818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.895746    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55vz\" (UniqueName: \"kubernetes.io/projected/b9805952-12b5-449d-a33c-bc3b089185e3-kube-api-access-f55vz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:38.895818 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:38.895775    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9805952-12b5-449d-a33c-bc3b089185e3-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:39.596135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.596063    4314 generic.go:356] "Generic (PLEG): container finished" podID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerID="b3ec34d0a0b4e9e58e53519249c69081318931e0047ec8aca290ee3827989dc4" exitCode=0
Nov 01 09:54:39.596309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.596140    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" event={"ID":"e29daa32-340d-4b9f-a146-c32c26c3ec03","Type":"ContainerDied","Data":"b3ec34d0a0b4e9e58e53519249c69081318931e0047ec8aca290ee3827989dc4"}
Nov 01 09:54:39.596309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.596245    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk" event={"ID":"e29daa32-340d-4b9f-a146-c32c26c3ec03","Type":"ContainerDied","Data":"1c8aa30613b3bae60a214959dff22a2c6bf1e147031baf7b1aae59aa495f2e76"}
Nov 01 09:54:39.596309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.596296    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c8aa30613b3bae60a214959dff22a2c6bf1e147031baf7b1aae59aa495f2e76"
Nov 01 09:54:39.596487 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.596403    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 09:54:39.598164 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.598139    4314 generic.go:356] "Generic (PLEG): container finished" podID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerID="d1190265b9e3ddcc1ac199e9a8fc19abcb13794c15ef751bbaf70e9dc54002f7" exitCode=0
Nov 01 09:54:39.598301 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.598212    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" event={"ID":"53c5850c-149f-4021-aa38-43468f0e0dfe","Type":"ContainerDied","Data":"d1190265b9e3ddcc1ac199e9a8fc19abcb13794c15ef751bbaf70e9dc54002f7"}
Nov 01 09:54:39.599781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.599758    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz" event={"ID":"b9805952-12b5-449d-a33c-bc3b089185e3","Type":"ContainerDied","Data":"4fa3810d774011caa466e43212084b2421df15a91ae1125c544a2484299c03f4"}
Nov 01 09:54:39.599885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.599799    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa3810d774011caa466e43212084b2421df15a91ae1125c544a2484299c03f4"
Nov 01 09:54:39.599885 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.599862    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"
Nov 01 09:54:39.693153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.693119    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:39.705035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.705008    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume\") pod \"e29daa32-340d-4b9f-a146-c32c26c3ec03\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") "
Nov 01 09:54:39.705225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.705071    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn\" (UniqueName: \"kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn\") pod \"e29daa32-340d-4b9f-a146-c32c26c3ec03\" (UID: \"e29daa32-340d-4b9f-a146-c32c26c3ec03\") "
Nov 01 09:54:39.705578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.705418    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume" pod "e29daa32-340d-4b9f-a146-c32c26c3ec03" (UID: "e29daa32-340d-4b9f-a146-c32c26c3ec03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:39.705713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.705624    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29daa32-340d-4b9f-a146-c32c26c3ec03-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:39.722798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.722764    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn" pod "e29daa32-340d-4b9f-a146-c32c26c3ec03" (UID: "e29daa32-340d-4b9f-a146-c32c26c3ec03"). InnerVolumeSpecName "kube-api-access-kncbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:39.777057 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.777033    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 09:54:39.806586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.806556    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb\" (UniqueName: \"kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb\") pod \"53c5850c-149f-4021-aa38-43468f0e0dfe\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") "
Nov 01 09:54:39.806734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.806600    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume\") pod \"53c5850c-149f-4021-aa38-43468f0e0dfe\" (UID: \"53c5850c-149f-4021-aa38-43468f0e0dfe\") "
Nov 01 09:54:39.807016 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.806995    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume" pod "53c5850c-149f-4021-aa38-43468f0e0dfe" (UID: "53c5850c-149f-4021-aa38-43468f0e0dfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:39.807105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.807067    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kncbn\" (UniqueName: \"kubernetes.io/projected/e29daa32-340d-4b9f-a146-c32c26c3ec03-kube-api-access-kncbn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:39.808627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.808606    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb" pod "53c5850c-149f-4021-aa38-43468f0e0dfe" (UID: "53c5850c-149f-4021-aa38-43468f0e0dfe"). InnerVolumeSpecName "kube-api-access-zwzmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:39.856327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.856174    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:39.908227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.908198    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzmb\" (UniqueName: \"kubernetes.io/projected/53c5850c-149f-4021-aa38-43468f0e0dfe-kube-api-access-zwzmb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:39.908227 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:39.908224    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5850c-149f-4021-aa38-43468f0e0dfe-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:40.606998 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.606970    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"
Nov 01 09:54:40.607204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.606998    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k" event={"ID":"53c5850c-149f-4021-aa38-43468f0e0dfe","Type":"ContainerDied","Data":"a5a0ae9b74a5cdc5e6e19b99b6e47c9b0b5752067f47ef9496640576008b6d39"}
Nov 01 09:54:40.607204 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.607052    4314 scope.go:122] "RemoveContainer" containerID="d1190265b9e3ddcc1ac199e9a8fc19abcb13794c15ef751bbaf70e9dc54002f7"
Nov 01 09:54:40.607343 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.607218    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"
Nov 01 09:54:40.619647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.619623    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:40.763785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.763754    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"]
Nov 01 09:54:40.766790 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.766763    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-54e01b-6d874c68ff-6gfpk"]
Nov 01 09:54:40.870295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.870194    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"]
Nov 01 09:54:40.874696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.874672    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-54e01b-767854c5f-k7z8k"]
Nov 01 09:54:40.929015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.928989    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" path="/var/lib/kubelet/pods/53c5850c-149f-4021-aa38-43468f0e0dfe/volumes"
Nov 01 09:54:40.929374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:40.929361    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" path="/var/lib/kubelet/pods/e29daa32-340d-4b9f-a146-c32c26c3ec03/volumes"
Nov 01 09:54:41.323274 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:54:41.323239    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:54:42.253247 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:42.253216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:46.380781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.380745    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"]
Nov 01 09:54:46.381348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381333    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b9805952-12b5-449d-a33c-bc3b089185e3" containerName="revoke-keys"
Nov 01 09:54:46.381400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381348    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9805952-12b5-449d-a33c-bc3b089185e3" containerName="revoke-keys"
Nov 01 09:54:46.381400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381370    4314 cpu_manager.go:414] "containerMap: removing container" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerName="payload-pre-processing"
Nov 01 09:54:46.381400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381376    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerName="payload-pre-processing"
Nov 01 09:54:46.381400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381397    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerName="payload-processing"
Nov 01 09:54:46.381546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381402    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerName="payload-processing"
Nov 01 09:54:46.381546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381476    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9805952-12b5-449d-a33c-bc3b089185e3" containerName="revoke-keys"
Nov 01 09:54:46.381546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381487    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c5850c-149f-4021-aa38-43468f0e0dfe" containerName="payload-pre-processing"
Nov 01 09:54:46.381546 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.381497    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29daa32-340d-4b9f-a146-c32c26c3ec03" containerName="payload-processing"
Nov 01 09:54:46.386415 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.386396    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.437913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.437885    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:54:46.437913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.437886    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:54:46.478287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.478243    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4bn\" (UniqueName: \"kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.478464 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.478416    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.478549 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.478499    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.489568 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.489520    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"]
Nov 01 09:54:46.508786 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.508759    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:54:46.508960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.508848    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:54:46.580059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.580032    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.580221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.580090    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.580221 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.580181    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4bn\" (UniqueName: \"kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.580853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.580829    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.582438 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.582416    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.641179 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.641119    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4bn\" (UniqueName: \"kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn\") pod \"maas-api-revoke-sub-ba313e68-dmhlm\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:46.700185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:46.700152    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:47.068252 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:47.068228    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"]
Nov 01 09:54:47.070171 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:54:47.070134    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d74a94e_a3e7_4185_99d6_c3503a217f26.slice/crio-8b7b71cdf01d79072e57403fdaf5cf47b335237ae771157b0cf0ac81c1f295a2 WatchSource:0}: Error finding container 8b7b71cdf01d79072e57403fdaf5cf47b335237ae771157b0cf0ac81c1f295a2: Status 404 returned error can't find the container with id 8b7b71cdf01d79072e57403fdaf5cf47b335237ae771157b0cf0ac81c1f295a2
Nov 01 09:54:47.245196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:47.245158    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:47.504174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:47.504144    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:47.655374 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:47.655341    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm" event={"ID":"2d74a94e-a3e7-4185-99d6-c3503a217f26","Type":"ContainerStarted","Data":"7952f6e0911999a624cd4aaffd8271aedf8925fc0bffe552fe419e3ddb0b338a"}
Nov 01 09:54:47.655555 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:47.655399    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm" event={"ID":"2d74a94e-a3e7-4185-99d6-c3503a217f26","Type":"ContainerStarted","Data":"8b7b71cdf01d79072e57403fdaf5cf47b335237ae771157b0cf0ac81c1f295a2"}
Nov 01 09:54:47.773641 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:47.773590    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm" podStartSLOduration=1.773575445 podStartE2EDuration="1.773575445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:54:47.769772674 +0000 UTC m=+4960.964167230" watchObservedRunningTime="2026-11-01 09:54:47.773575445 +0000 UTC m=+4960.967970038"
Nov 01 09:54:48.663664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:48.663632    4314 generic.go:356] "Generic (PLEG): container finished" podID="2d74a94e-a3e7-4185-99d6-c3503a217f26" containerID="7952f6e0911999a624cd4aaffd8271aedf8925fc0bffe552fe419e3ddb0b338a" exitCode=0
Nov 01 09:54:48.664085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:48.663698    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm" event={"ID":"2d74a94e-a3e7-4185-99d6-c3503a217f26","Type":"ContainerDied","Data":"7952f6e0911999a624cd4aaffd8271aedf8925fc0bffe552fe419e3ddb0b338a"}
Nov 01 09:54:49.826980 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.826954    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:49.918131 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.918096    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn\" (UniqueName: \"kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn\") pod \"2d74a94e-a3e7-4185-99d6-c3503a217f26\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") "
Nov 01 09:54:49.918307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.918142    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca\") pod \"2d74a94e-a3e7-4185-99d6-c3503a217f26\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") "
Nov 01 09:54:49.918307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.918180    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca\") pod \"2d74a94e-a3e7-4185-99d6-c3503a217f26\" (UID: \"2d74a94e-a3e7-4185-99d6-c3503a217f26\") "
Nov 01 09:54:49.918626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.918603    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca" pod "2d74a94e-a3e7-4185-99d6-c3503a217f26" (UID: "2d74a94e-a3e7-4185-99d6-c3503a217f26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:54:49.920425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.920361    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn" pod "2d74a94e-a3e7-4185-99d6-c3503a217f26" (UID: "2d74a94e-a3e7-4185-99d6-c3503a217f26"). InnerVolumeSpecName "kube-api-access-pp4bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:49.920425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:49.920360    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca" pod "2d74a94e-a3e7-4185-99d6-c3503a217f26" (UID: "2d74a94e-a3e7-4185-99d6-c3503a217f26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:50.019951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:50.019917    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4bn\" (UniqueName: \"kubernetes.io/projected/2d74a94e-a3e7-4185-99d6-c3503a217f26-kube-api-access-pp4bn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:50.019951 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:50.019953    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:50.020165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:50.019970    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d74a94e-a3e7-4185-99d6-c3503a217f26-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:50.677091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:50.677057    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"
Nov 01 09:54:50.677091 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:50.677082    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm" event={"ID":"2d74a94e-a3e7-4185-99d6-c3503a217f26","Type":"ContainerDied","Data":"8b7b71cdf01d79072e57403fdaf5cf47b335237ae771157b0cf0ac81c1f295a2"}
Nov 01 09:54:50.677308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:50.677131    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7b71cdf01d79072e57403fdaf5cf47b335237ae771157b0cf0ac81c1f295a2"
Nov 01 09:54:52.103147 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:52.103116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:53.784371 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:53.784316    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:54.760817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:54.760782    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:55.593316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:55.593274    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:55.780974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:55.780942    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:56.221279 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:56.221208    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:54:57.202928 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:54:57.202888    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:02.182916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:02.182816    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:04.337783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:04.337752    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:05.460190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:05.460157    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"]
Nov 01 09:55:05.466317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:05.466285    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:06.630802 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:06.630764    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:08.400559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:08.400500    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:10.996922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:10.996891    4314 scope.go:122] "RemoveContainer" containerID="37043857a7ead30de9cb8e84e0075b3fb3028aa1113ae2b288a402d0606369fd"
Nov 01 09:55:11.006742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.006715    4314 scope.go:122] "RemoveContainer" containerID="b3ec34d0a0b4e9e58e53519249c69081318931e0047ec8aca290ee3827989dc4"
Nov 01 09:55:11.015338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.015320    4314 scope.go:122] "RemoveContainer" containerID="cfa68f01f0249a7c7e1eacc7bedab4197761dcc9e04834d8dfad5222b19a53c4"
Nov 01 09:55:11.324118 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:55:11.324007    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:55:11.350733 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.350699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:11.961090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.960514    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"]
Nov 01 09:55:11.961649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.961628    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2d74a94e-a3e7-4185-99d6-c3503a217f26" containerName="revoke-keys"
Nov 01 09:55:11.961649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.961648    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d74a94e-a3e7-4185-99d6-c3503a217f26" containerName="revoke-keys"
Nov 01 09:55:11.961846 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.961734    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d74a94e-a3e7-4185-99d6-c3503a217f26" containerName="revoke-keys"
Nov 01 09:55:11.966671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:11.966649    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.016095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.016050    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:55:12.020512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.020482    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"]
Nov 01 09:55:12.062087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.062023    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:55:12.062087 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.062039    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:55:12.062393 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.062167    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:55:12.093439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.093413    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.093616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.093587    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.093677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.093624    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkx2\" (UniqueName: \"kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.194412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.194370    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.194629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.194427    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkx2\" (UniqueName: \"kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.194629 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.194497    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.195013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.194993    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.196807 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.196790    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.256649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.256622    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkx2\" (UniqueName: \"kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2\") pod \"maas-api-revoke-sub-bc190bf5-8md9r\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.281439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.281415    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:12.656684 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:55:12.656643    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ea3731_f49e_4700_9df0_0366e462b19a.slice/crio-abea5778d5f4d255564965b73542f9d79585b9fb317557aafdb20e0abb906a4d WatchSource:0}: Error finding container abea5778d5f4d255564965b73542f9d79585b9fb317557aafdb20e0abb906a4d: Status 404 returned error can't find the container with id abea5778d5f4d255564965b73542f9d79585b9fb317557aafdb20e0abb906a4d
Nov 01 09:55:12.656829 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.656756    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"]
Nov 01 09:55:12.821411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:12.821380    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r" event={"ID":"e5ea3731-f49e-4700-9df0-0366e462b19a","Type":"ContainerStarted","Data":"abea5778d5f4d255564965b73542f9d79585b9fb317557aafdb20e0abb906a4d"}
Nov 01 09:55:13.387502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:13.387466    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:13.829000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:13.828960    4314 generic.go:356] "Generic (PLEG): container finished" podID="e5ea3731-f49e-4700-9df0-0366e462b19a" containerID="f9ebec0fca3bc5d19d91e281de0c95b8cfda5e825e935bdec8978d375e0560b4" exitCode=0
Nov 01 09:55:13.829208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:13.829041    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r" event={"ID":"e5ea3731-f49e-4700-9df0-0366e462b19a","Type":"ContainerDied","Data":"f9ebec0fca3bc5d19d91e281de0c95b8cfda5e825e935bdec8978d375e0560b4"}
Nov 01 09:55:14.341586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:14.341557    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:14.499587 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:14.499551    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:14.685291 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:14.685216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:14.984281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:14.984254    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:15.130275 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.130241    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2\" (UniqueName: \"kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2\") pod \"e5ea3731-f49e-4700-9df0-0366e462b19a\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") "
Nov 01 09:55:15.130500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.130306    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca\") pod \"e5ea3731-f49e-4700-9df0-0366e462b19a\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") "
Nov 01 09:55:15.130500 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.130406    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca\") pod \"e5ea3731-f49e-4700-9df0-0366e462b19a\" (UID: \"e5ea3731-f49e-4700-9df0-0366e462b19a\") "
Nov 01 09:55:15.130754 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.130729    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca" pod "e5ea3731-f49e-4700-9df0-0366e462b19a" (UID: "e5ea3731-f49e-4700-9df0-0366e462b19a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:55:15.131018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.130999    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:15.132521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.132494    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca" pod "e5ea3731-f49e-4700-9df0-0366e462b19a" (UID: "e5ea3731-f49e-4700-9df0-0366e462b19a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:15.132705 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.132613    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2" pod "e5ea3731-f49e-4700-9df0-0366e462b19a" (UID: "e5ea3731-f49e-4700-9df0-0366e462b19a"). InnerVolumeSpecName "kube-api-access-7mkx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:15.232026 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.232001    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mkx2\" (UniqueName: \"kubernetes.io/projected/e5ea3731-f49e-4700-9df0-0366e462b19a-kube-api-access-7mkx2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:15.232175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.232029    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ea3731-f49e-4700-9df0-0366e462b19a-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:15.779260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.779224    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"]
Nov 01 09:55:15.844611 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.844578    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r" event={"ID":"e5ea3731-f49e-4700-9df0-0366e462b19a","Type":"ContainerDied","Data":"abea5778d5f4d255564965b73542f9d79585b9fb317557aafdb20e0abb906a4d"}
Nov 01 09:55:15.844771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.844626    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abea5778d5f4d255564965b73542f9d79585b9fb317557aafdb20e0abb906a4d"
Nov 01 09:55:15.844771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.844591    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"
Nov 01 09:55:15.851184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:15.851148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:16.326123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:16.326078    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"]
Nov 01 09:55:16.327340 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:16.327309    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:16.331775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:16.331751    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"]
Nov 01 09:55:16.339290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:16.339267    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"]
Nov 01 09:55:16.499728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:16.499693    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:16.832345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:16.832310    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:20.389208 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.389128    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:20.480280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.480221    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" podUID="826f3367-0030-46d6-8a81-d94de3680de8" containerName="main" containerID="cri-o://8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613" gracePeriod=45
Nov 01 09:55:20.556853 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.556822    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" podUID="826f3367-0030-46d6-8a81-d94de3680de8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.80:8000/ready\": dial tcp 10.132.1.80:8000: connect: connection refused"
Nov 01 09:55:20.743703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.743668    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:55:20.881260 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.881227    4314 generic.go:356] "Generic (PLEG): container finished" podID="826f3367-0030-46d6-8a81-d94de3680de8" containerID="8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613" exitCode=0
Nov 01 09:55:20.881458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.881292    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" event={"ID":"826f3367-0030-46d6-8a81-d94de3680de8","Type":"ContainerDied","Data":"8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613"}
Nov 01 09:55:20.881458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.881323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t" event={"ID":"826f3367-0030-46d6-8a81-d94de3680de8","Type":"ContainerDied","Data":"73965629d95ddc8fa614e8412c81eec8aa48788b10e105bcd544c98be6dcd03f"}
Nov 01 09:55:20.881458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.881339    4314 scope.go:122] "RemoveContainer" containerID="8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613"
Nov 01 09:55:20.881458 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.881295    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"
Nov 01 09:55:20.891586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.891516    4314 scope.go:122] "RemoveContainer" containerID="8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613"
Nov 01 09:55:20.891927 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:55:20.891906    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613\": container with ID starting with 8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613 not found: ID does not exist" containerID="8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613"
Nov 01 09:55:20.892007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.891937    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613"} err="failed to get container status \"8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613\": rpc error: code = NotFound desc = could not find container \"8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613\": container with ID starting with 8271f75ba81bfca748bdbdc625ec78b932a962d1cea873c549aa6fa8596ff613 not found: ID does not exist"
Nov 01 09:55:20.902313 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902292    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7\" (UniqueName: \"kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7\") pod \"826f3367-0030-46d6-8a81-d94de3680de8\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") "
Nov 01 09:55:20.902420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902331    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache\") pod \"826f3367-0030-46d6-8a81-d94de3680de8\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") "
Nov 01 09:55:20.902499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902437    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir\") pod \"826f3367-0030-46d6-8a81-d94de3680de8\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") "
Nov 01 09:55:20.902499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902456    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home\") pod \"826f3367-0030-46d6-8a81-d94de3680de8\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") "
Nov 01 09:55:20.902499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902493    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs\" (UniqueName: \"kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs\") pod \"826f3367-0030-46d6-8a81-d94de3680de8\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") "
Nov 01 09:55:20.902697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902629    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm\") pod \"826f3367-0030-46d6-8a81-d94de3680de8\" (UID: \"826f3367-0030-46d6-8a81-d94de3680de8\") "
Nov 01 09:55:20.902762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902705    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache" pod "826f3367-0030-46d6-8a81-d94de3680de8" (UID: "826f3367-0030-46d6-8a81-d94de3680de8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:20.902762 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902711    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir" pod "826f3367-0030-46d6-8a81-d94de3680de8" (UID: "826f3367-0030-46d6-8a81-d94de3680de8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:20.902871 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.902752    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home" pod "826f3367-0030-46d6-8a81-d94de3680de8" (UID: "826f3367-0030-46d6-8a81-d94de3680de8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:20.903127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.903106    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:20.903127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.903127    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:20.903307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.903136    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:20.904652 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.904633    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7" pod "826f3367-0030-46d6-8a81-d94de3680de8" (UID: "826f3367-0030-46d6-8a81-d94de3680de8"). InnerVolumeSpecName "kube-api-access-8lsz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:20.904727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.904671    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs" pod "826f3367-0030-46d6-8a81-d94de3680de8" (UID: "826f3367-0030-46d6-8a81-d94de3680de8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:20.904773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:20.904743    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm" pod "826f3367-0030-46d6-8a81-d94de3680de8" (UID: "826f3367-0030-46d6-8a81-d94de3680de8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:21.004745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:21.004714    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826f3367-0030-46d6-8a81-d94de3680de8-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:21.004745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:21.004740    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826f3367-0030-46d6-8a81-d94de3680de8-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:21.004745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:21.004749    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsz7\" (UniqueName: \"kubernetes.io/projected/826f3367-0030-46d6-8a81-d94de3680de8-kube-api-access-8lsz7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:21.289718 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:21.289689    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"]
Nov 01 09:55:21.295710 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:21.295680    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/rate-test-model-a87a1d-kserve-77b4b95cf7-kxp8t"]
Nov 01 09:55:22.588123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:22.588092    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:22.928412 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:22.928321    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826f3367-0030-46d6-8a81-d94de3680de8" path="/var/lib/kubelet/pods/826f3367-0030-46d6-8a81-d94de3680de8/volumes"
Nov 01 09:55:23.793528 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:23.793492    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:28.635512 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.635475    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"]
Nov 01 09:55:28.636056 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.636038    4314 cpu_manager.go:414] "containerMap: removing container" podUID="826f3367-0030-46d6-8a81-d94de3680de8" containerName="main"
Nov 01 09:55:28.636104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.636057    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826f3367-0030-46d6-8a81-d94de3680de8" containerName="main"
Nov 01 09:55:28.636104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.636098    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e5ea3731-f49e-4700-9df0-0366e462b19a" containerName="revoke-keys"
Nov 01 09:55:28.636104 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.636103    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ea3731-f49e-4700-9df0-0366e462b19a" containerName="revoke-keys"
Nov 01 09:55:28.636253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.636188    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ea3731-f49e-4700-9df0-0366e462b19a" containerName="revoke-keys"
Nov 01 09:55:28.636253 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.636198    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="826f3367-0030-46d6-8a81-d94de3680de8" containerName="main"
Nov 01 09:55:28.639773 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.639756    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.656566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.656514    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:28.696047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.696017    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"]
Nov 01 09:55:28.696794 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.696764    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.696934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.696821    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.697004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.696934    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbbl6\" (UniqueName: \"kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.737613 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.737578    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:55:28.737760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.737654    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:55:28.737760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.737748    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:55:28.740006 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.739989    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:55:28.798589 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.798554    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.798757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.798605    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.798757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.798652    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbbl6\" (UniqueName: \"kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.799192 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.799169    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.800960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.800938    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.828914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.828891    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:28.856447 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.856425    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbbl6\" (UniqueName: \"kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6\") pod \"maas-api-revoke-sub-e2c3e88d-l8bcw\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:28.954140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:28.953744    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:29.341609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:29.341580    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"]
Nov 01 09:55:29.343148 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:55:29.343123    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085c4146_9cce_4ab1_8797_e58847d8fb1f.slice/crio-103d9076a22ae83fb9e958bb0f1b88470463acf7d29ecc200792acec150138c5 WatchSource:0}: Error finding container 103d9076a22ae83fb9e958bb0f1b88470463acf7d29ecc200792acec150138c5: Status 404 returned error can't find the container with id 103d9076a22ae83fb9e958bb0f1b88470463acf7d29ecc200792acec150138c5
Nov 01 09:55:29.958508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:29.958421    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw" event={"ID":"085c4146-9cce-4ab1-8797-e58847d8fb1f","Type":"ContainerStarted","Data":"be2ced4bf9aee5bfc34307ccc76cc732286c9caf72b68f875cca00365d22ab19"}
Nov 01 09:55:29.958508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:29.958502    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw" event={"ID":"085c4146-9cce-4ab1-8797-e58847d8fb1f","Type":"ContainerStarted","Data":"103d9076a22ae83fb9e958bb0f1b88470463acf7d29ecc200792acec150138c5"}
Nov 01 09:55:30.072200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.072147    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw" podStartSLOduration=2.072133112 podStartE2EDuration="2.072133112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:55:30.069759355 +0000 UTC m=+5003.264153911" watchObservedRunningTime="2026-11-01 09:55:30.072133112 +0000 UTC m=+5003.266527667"
Nov 01 09:55:30.227756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.227727    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:30.423550 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.418256    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:30.799734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.799691    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" podUID="554f1092-3305-461f-9ed5-b29740798074" containerName="main" containerID="cri-o://46b8b734f476a1a852f1aff36be161105bb740ed26682f72a7d6c8c26b32b51e" gracePeriod=45
Nov 01 09:55:30.966199 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.966153    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" podUID="554f1092-3305-461f-9ed5-b29740798074" containerName="main" probeResult="failure" output="Get \"https://10.132.1.81:8000/ready\": dial tcp 10.132.1.81:8000: connect: connection refused"
Nov 01 09:55:30.966691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.966622    4314 generic.go:356] "Generic (PLEG): container finished" podID="554f1092-3305-461f-9ed5-b29740798074" containerID="46b8b734f476a1a852f1aff36be161105bb740ed26682f72a7d6c8c26b32b51e" exitCode=0
Nov 01 09:55:30.966776 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.966708    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" event={"ID":"554f1092-3305-461f-9ed5-b29740798074","Type":"ContainerDied","Data":"46b8b734f476a1a852f1aff36be161105bb740ed26682f72a7d6c8c26b32b51e"}
Nov 01 09:55:30.968439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.968417    4314 generic.go:356] "Generic (PLEG): container finished" podID="085c4146-9cce-4ab1-8797-e58847d8fb1f" containerID="be2ced4bf9aee5bfc34307ccc76cc732286c9caf72b68f875cca00365d22ab19" exitCode=0
Nov 01 09:55:30.968582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:30.968476    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw" event={"ID":"085c4146-9cce-4ab1-8797-e58847d8fb1f","Type":"ContainerDied","Data":"be2ced4bf9aee5bfc34307ccc76cc732286c9caf72b68f875cca00365d22ab19"}
Nov 01 09:55:31.066436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.066379    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:55:31.129850 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.129822    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b\" (UniqueName: \"kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b\") pod \"554f1092-3305-461f-9ed5-b29740798074\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") "
Nov 01 09:55:31.130055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.129856    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm\") pod \"554f1092-3305-461f-9ed5-b29740798074\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") "
Nov 01 09:55:31.130055 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.129988    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache\") pod \"554f1092-3305-461f-9ed5-b29740798074\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") "
Nov 01 09:55:31.130198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130088    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs\" (UniqueName: \"kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs\") pod \"554f1092-3305-461f-9ed5-b29740798074\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") "
Nov 01 09:55:31.130198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130132    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir\") pod \"554f1092-3305-461f-9ed5-b29740798074\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") "
Nov 01 09:55:31.130331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130227    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache" pod "554f1092-3305-461f-9ed5-b29740798074" (UID: "554f1092-3305-461f-9ed5-b29740798074"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.130331 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130267    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home\") pod \"554f1092-3305-461f-9ed5-b29740798074\" (UID: \"554f1092-3305-461f-9ed5-b29740798074\") "
Nov 01 09:55:31.130504 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130478    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir" pod "554f1092-3305-461f-9ed5-b29740798074" (UID: "554f1092-3305-461f-9ed5-b29740798074"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.130647 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130621    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home" pod "554f1092-3305-461f-9ed5-b29740798074" (UID: "554f1092-3305-461f-9ed5-b29740798074"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.130942 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130917    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.131020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130950    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.131020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.130968    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.132165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.132132    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b" pod "554f1092-3305-461f-9ed5-b29740798074" (UID: "554f1092-3305-461f-9ed5-b29740798074"). InnerVolumeSpecName "kube-api-access-qqk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:31.132165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.132132    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm" pod "554f1092-3305-461f-9ed5-b29740798074" (UID: "554f1092-3305-461f-9ed5-b29740798074"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.132324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.132191    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs" pod "554f1092-3305-461f-9ed5-b29740798074" (UID: "554f1092-3305-461f-9ed5-b29740798074"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:31.232114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.232078    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554f1092-3305-461f-9ed5-b29740798074-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.232114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.232103    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqk6b\" (UniqueName: \"kubernetes.io/projected/554f1092-3305-461f-9ed5-b29740798074-kube-api-access-qqk6b\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.232114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.232114    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554f1092-3305-461f-9ed5-b29740798074-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.347653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.347566    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main" containerID="cri-o://f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93" gracePeriod=45
Nov 01 09:55:31.352748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.352718    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main" containerID="cri-o://8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f" gracePeriod=45
Nov 01 09:55:31.359924 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.359892    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" containerID="cri-o://da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696" gracePeriod=45
Nov 01 09:55:31.443691 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.443650    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused"
Nov 01 09:55:31.698698 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.698604    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:31.701461 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.701120    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:55:31.707958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.706796    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:55:31.741686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741659    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache\") pod \"858111e0-6d9c-4a65-8a50-ce5d1a448001\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") "
Nov 01 09:55:31.741882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741721    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm\") pod \"858111e0-6d9c-4a65-8a50-ce5d1a448001\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") "
Nov 01 09:55:31.741882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741779    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home\") pod \"858111e0-6d9c-4a65-8a50-ce5d1a448001\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") "
Nov 01 09:55:31.741882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741817    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm\" (UniqueName: \"kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm\") pod \"858111e0-6d9c-4a65-8a50-ce5d1a448001\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") "
Nov 01 09:55:31.741882 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741868    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir\") pod \"858111e0-6d9c-4a65-8a50-ce5d1a448001\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") "
Nov 01 09:55:31.742144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741919    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs\" (UniqueName: \"kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs\") pod \"858111e0-6d9c-4a65-8a50-ce5d1a448001\" (UID: \"858111e0-6d9c-4a65-8a50-ce5d1a448001\") "
Nov 01 09:55:31.742144 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.741990    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache" pod "858111e0-6d9c-4a65-8a50-ce5d1a448001" (UID: "858111e0-6d9c-4a65-8a50-ce5d1a448001"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.742609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.742585    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.742744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.742687    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir" pod "858111e0-6d9c-4a65-8a50-ce5d1a448001" (UID: "858111e0-6d9c-4a65-8a50-ce5d1a448001"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.742744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.742704    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home" pod "858111e0-6d9c-4a65-8a50-ce5d1a448001" (UID: "858111e0-6d9c-4a65-8a50-ce5d1a448001"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.744870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.744843    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm" pod "858111e0-6d9c-4a65-8a50-ce5d1a448001" (UID: "858111e0-6d9c-4a65-8a50-ce5d1a448001"). InnerVolumeSpecName "kube-api-access-hx6pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:31.745919 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.745891    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm" pod "858111e0-6d9c-4a65-8a50-ce5d1a448001" (UID: "858111e0-6d9c-4a65-8a50-ce5d1a448001"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.748182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.748153    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs" pod "858111e0-6d9c-4a65-8a50-ce5d1a448001" (UID: "858111e0-6d9c-4a65-8a50-ce5d1a448001"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:31.788073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.788044    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:55:31.843586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843526    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache\") pod \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") "
Nov 01 09:55:31.843783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843609    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home\") pod \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") "
Nov 01 09:55:31.843783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843687    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg\" (UniqueName: \"kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg\") pod \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") "
Nov 01 09:55:31.843783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843725    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm\") pod \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") "
Nov 01 09:55:31.843783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843748    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir\") pod \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") "
Nov 01 09:55:31.843783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843782    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs\") pod \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") "
Nov 01 09:55:31.844099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843831    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir\") pod \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") "
Nov 01 09:55:31.844099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843862    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home\") pod \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") "
Nov 01 09:55:31.844099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843878    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl\" (UniqueName: \"kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl\") pod \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\" (UID: \"2bd618d9-cd13-4ab2-998b-a23c6ac10545\") "
Nov 01 09:55:31.844099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843898    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache\") pod \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") "
Nov 01 09:55:31.844099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843925    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs\") pod \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") "
Nov 01 09:55:31.844099 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.843961    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm\") pod \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\" (UID: \"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e\") "
Nov 01 09:55:31.845988 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.845688    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home" pod "2bd618d9-cd13-4ab2-998b-a23c6ac10545" (UID: "2bd618d9-cd13-4ab2-998b-a23c6ac10545"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.846378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.846178    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache" pod "2bd618d9-cd13-4ab2-998b-a23c6ac10545" (UID: "2bd618d9-cd13-4ab2-998b-a23c6ac10545"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.846648 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.846565    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache" pod "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" (UID: "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.848448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.848000    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir" pod "2bd618d9-cd13-4ab2-998b-a23c6ac10545" (UID: "2bd618d9-cd13-4ab2-998b-a23c6ac10545"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.848798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.848570    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home" pod "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" (UID: "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.848935    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir" pod "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" (UID: "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849333    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849357    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6pm\" (UniqueName: \"kubernetes.io/projected/858111e0-6d9c-4a65-8a50-ce5d1a448001-kube-api-access-hx6pm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849376    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849391    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849405    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858111e0-6d9c-4a65-8a50-ce5d1a448001-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849419    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849430    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849442    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849456    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849470    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858111e0-6d9c-4a65-8a50-ce5d1a448001-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849505 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.849483    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.850920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.850361    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs" pod "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" (UID: "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:31.851973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.851933    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm" pod "2bd618d9-cd13-4ab2-998b-a23c6ac10545" (UID: "2bd618d9-cd13-4ab2-998b-a23c6ac10545"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.855196 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.855175    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs" pod "2bd618d9-cd13-4ab2-998b-a23c6ac10545" (UID: "2bd618d9-cd13-4ab2-998b-a23c6ac10545"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:31.855312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.855263    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm" pod "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" (UID: "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.855312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.855289    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg" pod "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" (UID: "17cd82c9-b12d-4ea9-a1e5-efa69255bc4e"). InnerVolumeSpecName "kube-api-access-4vnwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:31.857094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.857069    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl" pod "2bd618d9-cd13-4ab2-998b-a23c6ac10545" (UID: "2bd618d9-cd13-4ab2-998b-a23c6ac10545"). InnerVolumeSpecName "kube-api-access-kxqhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:31.950905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.950872    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vnwg\" (UniqueName: \"kubernetes.io/projected/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-kube-api-access-4vnwg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.950905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.950902    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd618d9-cd13-4ab2-998b-a23c6ac10545-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.950905 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.950911    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd618d9-cd13-4ab2-998b-a23c6ac10545-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.951235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.950920    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxqhl\" (UniqueName: \"kubernetes.io/projected/2bd618d9-cd13-4ab2-998b-a23c6ac10545-kube-api-access-kxqhl\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.951235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.950928    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.951235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.950936    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.979074 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.979041    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"
Nov 01 09:55:31.979576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.979066    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl" event={"ID":"554f1092-3305-461f-9ed5-b29740798074","Type":"ContainerDied","Data":"fdf75ca589b87f3e33919d51ba8aaf4ab0f2132c4d9f1f4dd01a0e3e54319000"}
Nov 01 09:55:31.979576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.979126    4314 scope.go:122] "RemoveContainer" containerID="46b8b734f476a1a852f1aff36be161105bb740ed26682f72a7d6c8c26b32b51e"
Nov 01 09:55:31.980989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.980935    4314 generic.go:356] "Generic (PLEG): container finished" podID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerID="8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f" exitCode=0
Nov 01 09:55:31.981069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.981013    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" event={"ID":"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e","Type":"ContainerDied","Data":"8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f"}
Nov 01 09:55:31.981069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.981043    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd" event={"ID":"17cd82c9-b12d-4ea9-a1e5-efa69255bc4e","Type":"ContainerDied","Data":"7f07e302a4047956b914c39c0acfc02f15e20adafde46ac9d4f550fef8315ff5"}
Nov 01 09:55:31.981193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.981106    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"
Nov 01 09:55:31.994366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.994341    4314 scope.go:122] "RemoveContainer" containerID="8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f"
Nov 01 09:55:31.996133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.995688    4314 generic.go:356] "Generic (PLEG): container finished" podID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerID="f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93" exitCode=0
Nov 01 09:55:31.996133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.995824    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" event={"ID":"858111e0-6d9c-4a65-8a50-ce5d1a448001","Type":"ContainerDied","Data":"f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93"}
Nov 01 09:55:31.996133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.995858    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf" event={"ID":"858111e0-6d9c-4a65-8a50-ce5d1a448001","Type":"ContainerDied","Data":"053ffe3cce81c04b0dd5e960f6168f4b85df8dd0a249cda46075b3f5fd727024"}
Nov 01 09:55:31.996133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:31.996073    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"
Nov 01 09:55:32.006058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.006036    4314 generic.go:356] "Generic (PLEG): container finished" podID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerID="da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696" exitCode=0
Nov 01 09:55:32.006167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.006096    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"
Nov 01 09:55:32.006167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.006129    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" event={"ID":"2bd618d9-cd13-4ab2-998b-a23c6ac10545","Type":"ContainerDied","Data":"da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696"}
Nov 01 09:55:32.006257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.006173    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n" event={"ID":"2bd618d9-cd13-4ab2-998b-a23c6ac10545","Type":"ContainerDied","Data":"ef491f549ae3888b4db699eb03bae9a254ac37f9aa3e7cae30abb1691c3c1682"}
Nov 01 09:55:32.012621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.012598    4314 scope.go:122] "RemoveContainer" containerID="8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f"
Nov 01 09:55:32.012919 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:55:32.012898    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f\": container with ID starting with 8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f not found: ID does not exist" containerID="8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f"
Nov 01 09:55:32.012986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.012928    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f"} err="failed to get container status \"8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f\": rpc error: code = NotFound desc = could not find container \"8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f\": container with ID starting with 8e1dad797c2d5bdefc2b7b3e62d809d0a4be752eb9f38ef1f374ac16f26c770f not found: ID does not exist"
Nov 01 09:55:32.012986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.012946    4314 scope.go:122] "RemoveContainer" containerID="f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93"
Nov 01 09:55:32.025746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.025730    4314 scope.go:122] "RemoveContainer" containerID="f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93"
Nov 01 09:55:32.026189 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:55:32.026171    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93\": container with ID starting with f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93 not found: ID does not exist" containerID="f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93"
Nov 01 09:55:32.026295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.026278    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93"} err="failed to get container status \"f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93\": rpc error: code = NotFound desc = could not find container \"f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93\": container with ID starting with f5775713d23f3eeb58a54a3b4579c25a92762e6600160d832997af223b981a93 not found: ID does not exist"
Nov 01 09:55:32.026366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.026358    4314 scope.go:122] "RemoveContainer" containerID="da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696"
Nov 01 09:55:32.037409 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.037374    4314 scope.go:122] "RemoveContainer" containerID="da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696"
Nov 01 09:55:32.037986 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:55:32.037909    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696\": container with ID starting with da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696 not found: ID does not exist" containerID="da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696"
Nov 01 09:55:32.037986 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.037945    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696"} err="failed to get container status \"da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696\": rpc error: code = NotFound desc = could not find container \"da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696\": container with ID starting with da1492af6557c662ec7a88dd38fcc87e7f86f831266094abebe615ab2ca35696 not found: ID does not exist"
Nov 01 09:55:32.170715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.170681    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:32.172216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.172188    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"]
Nov 01 09:55:32.181783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.181755    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-343207/premium-w2-343207-kserve-cdb7bb78f-zrrdd"]
Nov 01 09:55:32.254872 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.254841    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca\") pod \"085c4146-9cce-4ab1-8797-e58847d8fb1f\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") "
Nov 01 09:55:32.255023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.254947    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6\" (UniqueName: \"kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6\") pod \"085c4146-9cce-4ab1-8797-e58847d8fb1f\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") "
Nov 01 09:55:32.255023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.255000    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca\") pod \"085c4146-9cce-4ab1-8797-e58847d8fb1f\" (UID: \"085c4146-9cce-4ab1-8797-e58847d8fb1f\") "
Nov 01 09:55:32.255373 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.255352    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca" pod "085c4146-9cce-4ab1-8797-e58847d8fb1f" (UID: "085c4146-9cce-4ab1-8797-e58847d8fb1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:55:32.255564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.255528    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:32.257051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.257032    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6" pod "085c4146-9cce-4ab1-8797-e58847d8fb1f" (UID: "085c4146-9cce-4ab1-8797-e58847d8fb1f"). InnerVolumeSpecName "kube-api-access-nbbl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:32.257105 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.257050    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca" pod "085c4146-9cce-4ab1-8797-e58847d8fb1f" (UID: "085c4146-9cce-4ab1-8797-e58847d8fb1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:32.286876 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.286849    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"]
Nov 01 09:55:32.292800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.292776    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-343207/simulator-w2-343207-kserve-985dd9f64-d6t2n"]
Nov 01 09:55:32.356923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.356894    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbbl6\" (UniqueName: \"kubernetes.io/projected/085c4146-9cce-4ab1-8797-e58847d8fb1f-kube-api-access-nbbl6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:32.356923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.356921    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c4146-9cce-4ab1-8797-e58847d8fb1f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:32.396922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.396856    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"]
Nov 01 09:55:32.401967 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.401939    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-w2-343207-kserve-7658d67596-cb6nf"]
Nov 01 09:55:32.510366 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.510331    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"]
Nov 01 09:55:32.514296 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.514264    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/rate-test-model-65aeef-kserve-7b976df745-mt7nl"]
Nov 01 09:55:32.930671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.930639    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" path="/var/lib/kubelet/pods/17cd82c9-b12d-4ea9-a1e5-efa69255bc4e/volumes"
Nov 01 09:55:32.931083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.931070    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" path="/var/lib/kubelet/pods/2bd618d9-cd13-4ab2-998b-a23c6ac10545/volumes"
Nov 01 09:55:32.931456 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.931444    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554f1092-3305-461f-9ed5-b29740798074" path="/var/lib/kubelet/pods/554f1092-3305-461f-9ed5-b29740798074/volumes"
Nov 01 09:55:32.932149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:32.932134    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" path="/var/lib/kubelet/pods/858111e0-6d9c-4a65-8a50-ce5d1a448001/volumes"
Nov 01 09:55:33.017696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:33.017670    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"
Nov 01 09:55:33.017696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:33.017682    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw" event={"ID":"085c4146-9cce-4ab1-8797-e58847d8fb1f","Type":"ContainerDied","Data":"103d9076a22ae83fb9e958bb0f1b88470463acf7d29ecc200792acec150138c5"}
Nov 01 09:55:33.018094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:33.017722    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="103d9076a22ae83fb9e958bb0f1b88470463acf7d29ecc200792acec150138c5"
Nov 01 09:55:34.233784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:34.233743    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:36.737370 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:36.737339    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:36.936188 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:36.936130    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:38.048775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:38.048743    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:39.209764 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:39.209732    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:41.323307 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:55:41.323270    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:55:43.818070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:43.818030    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:44.097604 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:44.097486    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:49.000216 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:49.000186    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:52.602759 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:52.602725    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:52.784623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:52.784578    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:54.019913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:54.019884    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:54.059947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:54.059911    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"]
Nov 01 09:55:54.105906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:54.105875    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0076559e-5bn55"]
Nov 01 09:55:54.928170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:54.928128    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abe6ae1-7752-4075-bdfd-16ac2dcc4e01" path="/var/lib/kubelet/pods/1abe6ae1-7752-4075-bdfd-16ac2dcc4e01/volumes"
Nov 01 09:55:55.544130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:55.544097    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:55:58.315290 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:55:58.315259    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:00.092916 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:00.092881    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:01.826626 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:01.826586    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:03.395140 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395092    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"]
Nov 01 09:56:03.395677 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395662    4314 cpu_manager.go:414] "containerMap: removing container" podUID="554f1092-3305-461f-9ed5-b29740798074" containerName="main"
Nov 01 09:56:03.395738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395679    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554f1092-3305-461f-9ed5-b29740798074" containerName="main"
Nov 01 09:56:03.395738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395709    4314 cpu_manager.go:414] "containerMap: removing container" podUID="085c4146-9cce-4ab1-8797-e58847d8fb1f" containerName="revoke-keys"
Nov 01 09:56:03.395738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395714    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085c4146-9cce-4ab1-8797-e58847d8fb1f" containerName="revoke-keys"
Nov 01 09:56:03.395738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395723    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main"
Nov 01 09:56:03.395738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395728    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main"
Nov 01 09:56:03.395738 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395739    4314 cpu_manager.go:414] "containerMap: removing container" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395744    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395755    4314 cpu_manager.go:414] "containerMap: removing container" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395759    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395830    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="085c4146-9cce-4ab1-8797-e58847d8fb1f" containerName="revoke-keys"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395837    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd618d9-cd13-4ab2-998b-a23c6ac10545" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395845    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cd82c9-b12d-4ea9-a1e5-efa69255bc4e" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395852    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="858111e0-6d9c-4a65-8a50-ce5d1a448001" containerName="main"
Nov 01 09:56:03.395958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.395859    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="554f1092-3305-461f-9ed5-b29740798074" containerName="main"
Nov 01 09:56:03.399578 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.399560    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.495543 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495496    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.495806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495564    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.495806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495623    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.495806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495706    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdlc\" (UniqueName: \"kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.495806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495716    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:03.495806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495759    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.495806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495799    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"sub-test-model-a87a1d-kserve-self-signed-certs\""
Nov 01 09:56:03.496130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495842    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"default-dockercfg-6tkpr\""
Nov 01 09:56:03.496130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.495921    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.496130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.496027    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"openshift-service-ca.crt\""
Nov 01 09:56:03.496130 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.496056    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"kube-root-ca.crt\""
Nov 01 09:56:03.499375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.499345    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"]
Nov 01 09:56:03.597327 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597294    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdlc\" (UniqueName: \"kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597345    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597395    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597457    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597484    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597554 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597507    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597809    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597871    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.597971 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.597919    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.599775 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.599758    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.600095 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.600076    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.655514 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.655452    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdlc\" (UniqueName: \"kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc\") pod \"sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:03.712649 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:03.712617    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:04.081991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:04.081962    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"]
Nov 01 09:56:04.083036 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:56:04.083011    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode70b58ac_0d89_4d44_a0e7_2c4b787b72b3.slice/crio-81242ddfafa34e48769f01428939094d26c9c08dfc3482f39d616a8999b4277a WatchSource:0}: Error finding container 81242ddfafa34e48769f01428939094d26c9c08dfc3482f39d616a8999b4277a: Status 404 returned error can't find the container with id 81242ddfafa34e48769f01428939094d26c9c08dfc3482f39d616a8999b4277a
Nov 01 09:56:04.217154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:04.217122    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" event={"ID":"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3","Type":"ContainerStarted","Data":"415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0"}
Nov 01 09:56:04.217154 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:04.217162    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" event={"ID":"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3","Type":"ContainerStarted","Data":"81242ddfafa34e48769f01428939094d26c9c08dfc3482f39d616a8999b4277a"}
Nov 01 09:56:04.331335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:04.331226    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" podStartSLOduration=1.3312116010000001 podStartE2EDuration="1.331211601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:56:04.327986091 +0000 UTC m=+5037.522380640" watchObservedRunningTime="2026-11-01 09:56:04.331211601 +0000 UTC m=+5037.525606155"
Nov 01 09:56:04.712952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:04.712872    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:04.896241 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:04.896208    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:08.377632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:08.377600    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:09.099803 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:09.099762    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:09.292064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:09.292030    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:09.811614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:09.811581    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:10.666716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:10.666683    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:11.176925 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:11.176895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:11.323575 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:56:11.323457    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:56:13.713720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:13.713691    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:13.726070 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:13.726047    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:14.292918 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:14.292886    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 09:56:15.100935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:15.100895    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:15.366050 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:15.365961    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:19.993513 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:19.993481    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:24.752293 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:24.752255    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:26.943027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:26.942995    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:28.938411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:28.938372    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:29.595225 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:29.595164    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:33.490990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:33.490947    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:34.102460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:34.102422    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"]
Nov 01 09:56:34.496879 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:34.496845    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:34.641011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:34.640968    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:39.103268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.103224    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6" podUID="b071d463-a17c-4535-9a8e-cde5581d4275" containerName="payload-processing" containerID="cri-o://f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1" gracePeriod=25
Nov 01 09:56:39.385166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.385140    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:56:39.449465 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.449436    4314 generic.go:356] "Generic (PLEG): container finished" podID="b071d463-a17c-4535-9a8e-cde5581d4275" containerID="f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1" exitCode=0
Nov 01 09:56:39.449660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.449487    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"
Nov 01 09:56:39.449660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.449520    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6" event={"ID":"b071d463-a17c-4535-9a8e-cde5581d4275","Type":"ContainerDied","Data":"f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1"}
Nov 01 09:56:39.449660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.449601    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6" event={"ID":"b071d463-a17c-4535-9a8e-cde5581d4275","Type":"ContainerDied","Data":"ee7a07a9e0a9efaa3aa4b5e0a0171905dc773f71364a7288c59e86191d2730e5"}
Nov 01 09:56:39.449660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.449630    4314 scope.go:122] "RemoveContainer" containerID="f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1"
Nov 01 09:56:39.459234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.459213    4314 scope.go:122] "RemoveContainer" containerID="f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1"
Nov 01 09:56:39.459546 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:56:39.459513    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1\": container with ID starting with f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1 not found: ID does not exist" containerID="f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1"
Nov 01 09:56:39.459610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.459569    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1"} err="failed to get container status \"f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1\": rpc error: code = NotFound desc = could not find container \"f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1\": container with ID starting with f01ef808b9b8f20b025cdadbfdac8b7cd7260f90e11b12ba932e185d64ba78e1 not found: ID does not exist"
Nov 01 09:56:39.518798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.518765    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume\") pod \"b071d463-a17c-4535-9a8e-cde5581d4275\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") "
Nov 01 09:56:39.518974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.518863    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml\" (UniqueName: \"kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml\") pod \"b071d463-a17c-4535-9a8e-cde5581d4275\" (UID: \"b071d463-a17c-4535-9a8e-cde5581d4275\") "
Nov 01 09:56:39.519195 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.519172    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume" pod "b071d463-a17c-4535-9a8e-cde5581d4275" (UID: "b071d463-a17c-4535-9a8e-cde5581d4275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:56:39.521013 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.520982    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml" pod "b071d463-a17c-4535-9a8e-cde5581d4275" (UID: "b071d463-a17c-4535-9a8e-cde5581d4275"). InnerVolumeSpecName "kube-api-access-zztml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:56:39.609300 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.609218    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:39.620306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.620272    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zztml\" (UniqueName: \"kubernetes.io/projected/b071d463-a17c-4535-9a8e-cde5581d4275-kube-api-access-zztml\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:39.620306 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.620305    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b071d463-a17c-4535-9a8e-cde5581d4275-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:39.865391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.865307    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"]
Nov 01 09:56:39.911232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:39.911199    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0401bd23-7f677b7bc5-fsrt6"]
Nov 01 09:56:40.928351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:40.928321    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b071d463-a17c-4535-9a8e-cde5581d4275" path="/var/lib/kubelet/pods/b071d463-a17c-4535-9a8e-cde5581d4275/volumes"
Nov 01 09:56:41.078002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:41.077966    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:41.321764 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:56:41.321721    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:56:43.809935 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:43.809897    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:45.580250 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:45.580216    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:47.048073 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:47.048040    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:47.193402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:47.193366    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:48.643824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:48.643790    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:49.271694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:49.271663    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"]
Nov 01 09:56:49.271930 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:49.271907    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" containerID="cri-o://1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e" gracePeriod=30
Nov 01 09:56:49.290784 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:49.290751    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:49.467134 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:49.467093    4314 patch_prober.go:29] interesting pod/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:56:49.467315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:49.467159    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:56:50.062956 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:50.062921    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:50.643609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:50.643566    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:54.530817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.530784    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:56:54.545169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.545100    4314 generic.go:356] "Generic (PLEG): container finished" podID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerID="1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e" exitCode=0
Nov 01 09:56:54.545169 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.545162    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"
Nov 01 09:56:54.545332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.545203    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" event={"ID":"f9bf61f8-2793-493b-8ef8-5da66598ab2c","Type":"ContainerDied","Data":"1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e"}
Nov 01 09:56:54.545332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.545264    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb" event={"ID":"f9bf61f8-2793-493b-8ef8-5da66598ab2c","Type":"ContainerDied","Data":"07ea0dacbf5d4885f3eb98e4726a1207f5b3e13fb90c43e35a5714cf1b845927"}
Nov 01 09:56:54.545332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.545275    4314 scope.go:122] "RemoveContainer" containerID="1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e"
Nov 01 09:56:54.558392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.558372    4314 scope.go:122] "RemoveContainer" containerID="5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"
Nov 01 09:56:54.567484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.567467    4314 scope.go:122] "RemoveContainer" containerID="1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e"
Nov 01 09:56:54.567811 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:56:54.567790    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e\": container with ID starting with 1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e not found: ID does not exist" containerID="1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e"
Nov 01 09:56:54.567915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.567822    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e"} err="failed to get container status \"1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e\": rpc error: code = NotFound desc = could not find container \"1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e\": container with ID starting with 1f247c55a27cc0ebcd0c51ffedadebd2fedc04ffd2aa4cc652da898964dd883e not found: ID does not exist"
Nov 01 09:56:54.567915 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.567847    4314 scope.go:122] "RemoveContainer" containerID="5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"
Nov 01 09:56:54.568107 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:56:54.568087    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7\": container with ID starting with 5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7 not found: ID does not exist" containerID="5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"
Nov 01 09:56:54.568160 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.568119    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7"} err="failed to get container status \"5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7\": rpc error: code = NotFound desc = could not find container \"5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7\": container with ID starting with 5b9f5a5b0f393e3b7c4a145decea03ac9ae4b650a95dde0c5a915457a07528a7 not found: ID does not exist"
Nov 01 09:56:54.600995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.600967    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601016    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601069    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601164    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601212 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601197    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601269    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601337    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601350    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:54.601421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601389    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601553 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601432    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy\") pod \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\" (UID: \"f9bf61f8-2793-493b-8ef8-5da66598ab2c\") "
Nov 01 09:56:54.601614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601387    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:56:54.601614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601559    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:54.601614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601522    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:54.601936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601908    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:54.601936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601924    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.602121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601944    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.602121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601959    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.602121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.601972    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.603646 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.603619    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:54.603796 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.603773    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:56:54.603912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.603890    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "kube-api-access-9l6rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:56:54.604096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.604078    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo" pod "f9bf61f8-2793-493b-8ef8-5da66598ab2c" (UID: "f9bf61f8-2793-493b-8ef8-5da66598ab2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:56:54.703010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.702974    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.703010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.703006    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.703010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.703017    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l6rq\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-kube-api-access-9l6rq\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.703266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.703027    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bf61f8-2793-493b-8ef8-5da66598ab2c-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.703266 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.703036    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bf61f8-2793-493b-8ef8-5da66598ab2c-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:54.962947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.962913    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"]
Nov 01 09:56:54.967146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:54.967115    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0401bd23-openshift-default-588d77f74f-g2ctb"]
Nov 01 09:56:55.697976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:55.697940    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:56.086753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:56.086717    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:56.756653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:56.756615    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"]
Nov 01 09:56:56.854837 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:56.854803    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"]
Nov 01 09:56:56.928354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:56.928318    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" path="/var/lib/kubelet/pods/f9bf61f8-2793-493b-8ef8-5da66598ab2c/volumes"
Nov 01 09:56:57.111744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:57.111651    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:56:57.256198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:56:57.256162    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:00.206999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:00.206965    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:01.757097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:01.757048    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerName="payload-processing" containerID="cri-o://3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194" gracePeriod=25
Nov 01 09:57:01.856032 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:01.855993    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerName="payload-pre-processing" containerID="cri-o://471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14" gracePeriod=25
Nov 01 09:57:01.917694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:01.917661    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:02.131035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:02.130934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:03.267407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:03.267366    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:57:03.267870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:03.267638    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:57:03.545683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:03.545607    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:04.017271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:04.017236    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"]
Nov 01 09:57:04.017451 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:04.017430    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy" containerID="cri-o://334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f" gracePeriod=30
Nov 01 09:57:04.028058 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:04.028029    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:04.758034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:04.758003    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:05.992720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:05.992687    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:07.127926 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:07.127891    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:09.268624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.268597    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:57:09.376746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376668    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.376746 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376721    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.376954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376781    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.376954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376803    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.376954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376848    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.376954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376887    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.376954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376937    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.377223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.376981    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:09.377223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377025    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.377223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377121    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket\") pod \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\" (UID: \"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5\") "
Nov 01 09:57:09.377407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377240    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:09.377407 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377268    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:09.377561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377464    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:09.377561 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377477    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:09.377692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377664    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.377692 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377687    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.377813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377702    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.377813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377716    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.377813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.377728    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.379157 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.379129    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:57:09.379303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.379220    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:09.379303 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.379266    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "kube-api-access-68p8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:09.379398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.379305    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token" pod "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" (UID: "1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:09.479262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.479224    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.479262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.479254    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.479262 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.479266    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.479576 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.479279    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68p8t\" (UniqueName: \"kubernetes.io/projected/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5-kube-api-access-68p8t\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:09.643025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.642951    4314 generic.go:356] "Generic (PLEG): container finished" podID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerID="334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f" exitCode=0
Nov 01 09:57:09.643025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.642997    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"
Nov 01 09:57:09.643220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.643034    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" event={"ID":"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5","Type":"ContainerDied","Data":"334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f"}
Nov 01 09:57:09.643220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.643091    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk" event={"ID":"1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5","Type":"ContainerDied","Data":"23ee5dc16db5e368cfecd50de2370322cedf0979b86d1ede27c59e4d27e2774f"}
Nov 01 09:57:09.643220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.643115    4314 scope.go:122] "RemoveContainer" containerID="334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f"
Nov 01 09:57:09.654324 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.654302    4314 scope.go:122] "RemoveContainer" containerID="43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"
Nov 01 09:57:09.663760 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.663726    4314 scope.go:122] "RemoveContainer" containerID="334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f"
Nov 01 09:57:09.664027 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:09.664008    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f\": container with ID starting with 334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f not found: ID does not exist" containerID="334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f"
Nov 01 09:57:09.664075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.664035    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f"} err="failed to get container status \"334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f\": rpc error: code = NotFound desc = could not find container \"334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f\": container with ID starting with 334bc57f89eaf5c3a0f31d290327f43e4d96c089ddb3e5d1fda7baffe821048f not found: ID does not exist"
Nov 01 09:57:09.664075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.664052    4314 scope.go:122] "RemoveContainer" containerID="43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"
Nov 01 09:57:09.664294 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:09.664271    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce\": container with ID starting with 43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce not found: ID does not exist" containerID="43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"
Nov 01 09:57:09.664335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.664301    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce"} err="failed to get container status \"43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce\": rpc error: code = NotFound desc = could not find container \"43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce\": container with ID starting with 43d17b55e7551341a1d65be57fc68db5e5a0696390302fe565b62e503b0bb1ce not found: ID does not exist"
Nov 01 09:57:09.797122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.797090    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"]
Nov 01 09:57:09.801068 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.801039    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-58de5a-openshift-default-65c69bcc55-mqvnk"]
Nov 01 09:57:09.981493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:09.981457    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"]
Nov 01 09:57:10.249090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.249008    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:10.450398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450366    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"]
Nov 01 09:57:10.450891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450877    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy"
Nov 01 09:57:10.450950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450892    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy"
Nov 01 09:57:10.450950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450905    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b071d463-a17c-4535-9a8e-cde5581d4275" containerName="payload-processing"
Nov 01 09:57:10.450950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450910    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b071d463-a17c-4535-9a8e-cde5581d4275" containerName="payload-processing"
Nov 01 09:57:10.450950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450924    4314 cpu_manager.go:414] "containerMap: removing container" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy"
Nov 01 09:57:10.450950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450929    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy"
Nov 01 09:57:10.450950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450949    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450955    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450964    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.450969    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.451038    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.451045    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b071d463-a17c-4535-9a8e-cde5581d4275" containerName="payload-processing"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.451053    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.451062    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy"
Nov 01 09:57:10.451162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.451070    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf61f8-2793-493b-8ef8-5da66598ab2c" containerName="istio-proxy"
Nov 01 09:57:10.455866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.455849    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.506792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.506368    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"kube-root-ca.crt\""
Nov 01 09:57:10.506792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.506495    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"openshift-service-ca.crt\""
Nov 01 09:57:10.506792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.506677    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"sub-test-model-65aeef-kserve-self-signed-certs\""
Nov 01 09:57:10.507111 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.507072    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-65aeef\"/\"default-dockercfg-qvwzz\""
Nov 01 09:57:10.545671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.545623    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"]
Nov 01 09:57:10.555351 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.555300    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:10.592007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.591974    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.592172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.592019    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.592172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.592079    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.592172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.592112    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.592172 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.592144    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.592354 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.592318    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtz9\" (UniqueName: \"kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693253    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtz9\" (UniqueName: \"kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693325    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693345    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693360    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693380    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693422    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693839    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693861    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.693929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.693899    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.695722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.695701    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.696020 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.696002    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.750745 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.750719    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtz9\" (UniqueName: \"kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9\") pod \"sub-test-model-65aeef-kserve-5bcbc599c-kpzxb\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") " pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.768158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.768098    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:10.937083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:10.933638    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5" path="/var/lib/kubelet/pods/1910e0cc-1bef-4ef4-96cb-0b7ddc7167e5/volumes"
Nov 01 09:57:11.106406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.106331    4314 scope.go:122] "RemoveContainer" containerID="b2e041864295536ff5c3dd0795afd0df2977f78492816440cc05bf2311b2e267"
Nov 01 09:57:11.144713 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.144687    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"]
Nov 01 09:57:11.147019 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:57:11.146993    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfee9992c_2ff3_4033_86b6_e6303cc588e0.slice/crio-d3d3230745ae149912dde07237ac95aeccc4c80a27b885df63b03b422c2e9ccd WatchSource:0}: Error finding container d3d3230745ae149912dde07237ac95aeccc4c80a27b885df63b03b422c2e9ccd: Status 404 returned error can't find the container with id d3d3230745ae149912dde07237ac95aeccc4c80a27b885df63b03b422c2e9ccd
Nov 01 09:57:11.323714 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:11.323665    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:57:11.394751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.394674    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:11.671119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.671005    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb" event={"ID":"fee9992c-2ff3-4033-86b6-e6303cc588e0","Type":"ContainerStarted","Data":"6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5"}
Nov 01 09:57:11.671119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.671065    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb" event={"ID":"fee9992c-2ff3-4033-86b6-e6303cc588e0","Type":"ContainerStarted","Data":"d3d3230745ae149912dde07237ac95aeccc4c80a27b885df63b03b422c2e9ccd"}
Nov 01 09:57:11.772040 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.768927    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:11.786480 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.786407    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb" podStartSLOduration=1.782892539 podStartE2EDuration="1.782892539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:57:11.780105688 +0000 UTC m=+5104.974500260" watchObservedRunningTime="2026-11-01 09:57:11.782892539 +0000 UTC m=+5104.977287097"
Nov 01 09:57:11.818931 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:11.818896    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:12.286135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:12.286100    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:13.268155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:13.268105    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:57:13.268155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:13.268141    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:57:14.982039 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:14.981973    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87" podUID="d56ad473-a2a7-4da5-87ed-1929a0ee715b" containerName="payload-pre-processing" containerID="cri-o://985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d" gracePeriod=25
Nov 01 09:57:15.243783 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.243728    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:57:15.342257 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.342226    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume\") pod \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") "
Nov 01 09:57:15.342430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.342296    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp\") pod \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\" (UID: \"d56ad473-a2a7-4da5-87ed-1929a0ee715b\") "
Nov 01 09:57:15.342596 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.342575    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume" pod "d56ad473-a2a7-4da5-87ed-1929a0ee715b" (UID: "d56ad473-a2a7-4da5-87ed-1929a0ee715b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:15.342814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.342802    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56ad473-a2a7-4da5-87ed-1929a0ee715b-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:15.344423 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.344399    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp" pod "d56ad473-a2a7-4da5-87ed-1929a0ee715b" (UID: "d56ad473-a2a7-4da5-87ed-1929a0ee715b"). InnerVolumeSpecName "kube-api-access-r5qsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:15.444490 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.444460    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/d56ad473-a2a7-4da5-87ed-1929a0ee715b-kube-api-access-r5qsp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:15.695559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.695441    4314 generic.go:356] "Generic (PLEG): container finished" podID="d56ad473-a2a7-4da5-87ed-1929a0ee715b" containerID="985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d" exitCode=0
Nov 01 09:57:15.695559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.695494    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"
Nov 01 09:57:15.695559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.695527    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87" event={"ID":"d56ad473-a2a7-4da5-87ed-1929a0ee715b","Type":"ContainerDied","Data":"985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d"}
Nov 01 09:57:15.695826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.695602    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87" event={"ID":"d56ad473-a2a7-4da5-87ed-1929a0ee715b","Type":"ContainerDied","Data":"caf103ca0e4adb3b45dec7933bf7073321799b3de21e564f1cc2f99217ce97b0"}
Nov 01 09:57:15.695826 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.695627    4314 scope.go:122] "RemoveContainer" containerID="985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d"
Nov 01 09:57:15.706318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.706300    4314 scope.go:122] "RemoveContainer" containerID="985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d"
Nov 01 09:57:15.706613 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:15.706594    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d\": container with ID starting with 985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d not found: ID does not exist" containerID="985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d"
Nov 01 09:57:15.706730 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.706625    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d"} err="failed to get container status \"985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d\": rpc error: code = NotFound desc = could not find container \"985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d\": container with ID starting with 985d338166a8bec37a0c00f82db8979295b25a74d52014b118adfab49a42671d not found: ID does not exist"
Nov 01 09:57:15.849321 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.849284    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"]
Nov 01 09:57:15.855001 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:15.854972    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a83714-699c6d89f9-tqj87"]
Nov 01 09:57:16.928653 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:16.928623    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56ad473-a2a7-4da5-87ed-1929a0ee715b" path="/var/lib/kubelet/pods/d56ad473-a2a7-4da5-87ed-1929a0ee715b/volumes"
Nov 01 09:57:17.282334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:17.282285    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:17.526847 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:17.526816    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"]
Nov 01 09:57:17.527045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:17.527022    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy" containerID="cri-o://c08af440eaa276c7f542d9b0825c06b6fcef4d1e31528122d9396dfe53471de2" gracePeriod=30
Nov 01 09:57:17.573127 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:17.573044    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"]
Nov 01 09:57:17.573320 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:17.573292    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy" containerID="cri-o://d58b913daad100c8b6f27371319e22dbaeaf106c375c8ffbe67f493bb79d8f5d" gracePeriod=30
Nov 01 09:57:18.495475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:18.495440    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:20.768516 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:20.768484    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:20.780714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:20.780689    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:21.744858 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:21.744829    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 09:57:22.010665 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.010606    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:57:22.108308 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.108280    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:57:22.122614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.122583    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume\") pod \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") "
Nov 01 09:57:22.122768 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.122648    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs\" (UniqueName: \"kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs\") pod \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\" (UID: \"b485d1df-e0e6-46a9-87b2-6aa46a71c488\") "
Nov 01 09:57:22.124203 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.124174    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume" pod "b485d1df-e0e6-46a9-87b2-6aa46a71c488" (UID: "b485d1df-e0e6-46a9-87b2-6aa46a71c488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:22.125233 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.125204    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs" pod "b485d1df-e0e6-46a9-87b2-6aa46a71c488" (UID: "b485d1df-e0e6-46a9-87b2-6aa46a71c488"). InnerVolumeSpecName "kube-api-access-b59zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:22.171323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.171294    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:22.224953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.224914    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume\") pod \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") "
Nov 01 09:57:22.225174 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.225047    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv\" (UniqueName: \"kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv\") pod \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\" (UID: \"2122f13b-f67a-43bc-abcf-7e434a8b2eff\") "
Nov 01 09:57:22.225436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.225409    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume" pod "2122f13b-f67a-43bc-abcf-7e434a8b2eff" (UID: "2122f13b-f67a-43bc-abcf-7e434a8b2eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:22.225697 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.225675    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b485d1df-e0e6-46a9-87b2-6aa46a71c488-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.225785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.225703    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b59zs\" (UniqueName: \"kubernetes.io/projected/b485d1df-e0e6-46a9-87b2-6aa46a71c488-kube-api-access-b59zs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.225785 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.225717    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2122f13b-f67a-43bc-abcf-7e434a8b2eff-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.227426 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.227403    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv" pod "2122f13b-f67a-43bc-abcf-7e434a8b2eff" (UID: "2122f13b-f67a-43bc-abcf-7e434a8b2eff"). InnerVolumeSpecName "kube-api-access-b7qhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:22.326727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.326643    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qhv\" (UniqueName: \"kubernetes.io/projected/2122f13b-f67a-43bc-abcf-7e434a8b2eff-kube-api-access-b7qhv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.745286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.742846    4314 generic.go:356] "Generic (PLEG): container finished" podID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerID="471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14" exitCode=0
Nov 01 09:57:22.745286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.742973    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" event={"ID":"2122f13b-f67a-43bc-abcf-7e434a8b2eff","Type":"ContainerDied","Data":"471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14"}
Nov 01 09:57:22.745286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.743062    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2" event={"ID":"2122f13b-f67a-43bc-abcf-7e434a8b2eff","Type":"ContainerDied","Data":"0a9fd2edc2ee087d682c70e3950ecdfe764e267a8716b2db0257452962f9bf1f"}
Nov 01 09:57:22.745286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.743092    4314 scope.go:122] "RemoveContainer" containerID="471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14"
Nov 01 09:57:22.745286 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.743259    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"
Nov 01 09:57:22.752593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.752147    4314 generic.go:356] "Generic (PLEG): container finished" podID="32b83241-0e9e-4599-83c6-77ac7521427e" containerID="d58b913daad100c8b6f27371319e22dbaeaf106c375c8ffbe67f493bb79d8f5d" exitCode=0
Nov 01 09:57:22.752593 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.752235    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" event={"ID":"32b83241-0e9e-4599-83c6-77ac7521427e","Type":"ContainerDied","Data":"d58b913daad100c8b6f27371319e22dbaeaf106c375c8ffbe67f493bb79d8f5d"}
Nov 01 09:57:22.755559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.754830    4314 generic.go:356] "Generic (PLEG): container finished" podID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerID="3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194" exitCode=0
Nov 01 09:57:22.755559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.754987    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" event={"ID":"b485d1df-e0e6-46a9-87b2-6aa46a71c488","Type":"ContainerDied","Data":"3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194"}
Nov 01 09:57:22.755559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.755020    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr" event={"ID":"b485d1df-e0e6-46a9-87b2-6aa46a71c488","Type":"ContainerDied","Data":"aa92ee49e07d228192f803eed65f7d0d51eda348cee9849020264e69d3f0f05a"}
Nov 01 09:57:22.755559 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.755089    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"
Nov 01 09:57:22.768230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.765275    4314 generic.go:356] "Generic (PLEG): container finished" podID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerID="c08af440eaa276c7f542d9b0825c06b6fcef4d1e31528122d9396dfe53471de2" exitCode=0
Nov 01 09:57:22.768230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.765707    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" event={"ID":"8837ef1f-f84f-484d-89b4-49ef01386c8c","Type":"ContainerDied","Data":"c08af440eaa276c7f542d9b0825c06b6fcef4d1e31528122d9396dfe53471de2"}
Nov 01 09:57:22.768230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.766204    4314 scope.go:122] "RemoveContainer" containerID="471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14"
Nov 01 09:57:22.768230 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:22.766472    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14\": container with ID starting with 471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14 not found: ID does not exist" containerID="471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14"
Nov 01 09:57:22.768230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.766508    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14"} err="failed to get container status \"471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14\": rpc error: code = NotFound desc = could not find container \"471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14\": container with ID starting with 471d7326e0dc95d632d00e0d82c45cf2ad954fcbdbf719b51a0698610e6efc14 not found: ID does not exist"
Nov 01 09:57:22.768230 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.766556    4314 scope.go:122] "RemoveContainer" containerID="3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194"
Nov 01 09:57:22.789624 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.789601    4314 scope.go:122] "RemoveContainer" containerID="3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194"
Nov 01 09:57:22.789940 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:22.789918    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194\": container with ID starting with 3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194 not found: ID does not exist" containerID="3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194"
Nov 01 09:57:22.790007 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.789950    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194"} err="failed to get container status \"3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194\": rpc error: code = NotFound desc = could not find container \"3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194\": container with ID starting with 3894470ed064f7b0ba9430416838ce12e8f6ba04d4a09713aa78bda0d4d6f194 not found: ID does not exist"
Nov 01 09:57:22.859137 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.859067    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:57:22.901782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.901749    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:57:22.904963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.904940    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"]
Nov 01 09:57:22.936912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.936886    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.936979    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937009    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937046 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937030    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937051    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937110    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937226 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937135    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:22.937406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937221    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937301    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937325    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:22.937406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937339    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy\") pod \"8837ef1f-f84f-484d-89b4-49ef01386c8c\" (UID: \"8837ef1f-f84f-484d-89b4-49ef01386c8c\") "
Nov 01 09:57:22.937671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937420    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:22.937671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937559    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:22.937671 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937580    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:22.937891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937867    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.937972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937899    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.937972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937913    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.937972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937928    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8837ef1f-f84f-484d-89b4-49ef01386c8c-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.937972 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.937943    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:22.939669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.939647    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:22.939864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.939834    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:22.939864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.939842    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "kube-api-access-fp5tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:22.940085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.940065    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo" pod "8837ef1f-f84f-484d-89b4-49ef01386c8c" (UID: "8837ef1f-f84f-484d-89b4-49ef01386c8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:57:22.951263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:22.951242    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-58de5a-68cb8c85c5-sntq2"]
Nov 01 09:57:23.012511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.012481    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"]
Nov 01 09:57:23.016600 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.016575    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-58de5a-74949f95f7-8trwr"]
Nov 01 09:57:23.038358 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038335    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038382    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038425    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038497    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038524 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038516    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038581    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038612    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038632    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038659    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn\") pod \"32b83241-0e9e-4599-83c6-77ac7521427e\" (UID: \"32b83241-0e9e-4599-83c6-77ac7521427e\") "
Nov 01 09:57:23.038910 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038829    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:23.038981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038908    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:23.038981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.038968    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:23.039109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039058    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:23.039173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039149    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039173 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039169    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039183    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp5tr\" (UniqueName: \"kubernetes.io/projected/8837ef1f-f84f-484d-89b4-49ef01386c8c-kube-api-access-fp5tr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039199    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039214    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039227    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039240    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8837ef1f-f84f-484d-89b4-49ef01386c8c-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.039332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.039288    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:23.040938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.040917    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "kube-api-access-4r9nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:23.041037 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.041018    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:57:23.041138 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.041122    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:23.041193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.041130    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy" pod "32b83241-0e9e-4599-83c6-77ac7521427e" (UID: "32b83241-0e9e-4599-83c6-77ac7521427e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:23.140494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.140424    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.140494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.140449    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b83241-0e9e-4599-83c6-77ac7521427e-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.140494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.140460    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b83241-0e9e-4599-83c6-77ac7521427e-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.140494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.140468    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9nn\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-kube-api-access-4r9nn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.140494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.140478    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b83241-0e9e-4599-83c6-77ac7521427e-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.140494 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.140487    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b83241-0e9e-4599-83c6-77ac7521427e-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:23.774369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.774281    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"
Nov 01 09:57:23.774369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.774294    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8" event={"ID":"32b83241-0e9e-4599-83c6-77ac7521427e","Type":"ContainerDied","Data":"8500a300063d95b6fb83d1da8683a25bb499cb4042eb18531fc5de0160159c7a"}
Nov 01 09:57:23.774369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.774364    4314 scope.go:122] "RemoveContainer" containerID="d58b913daad100c8b6f27371319e22dbaeaf106c375c8ffbe67f493bb79d8f5d"
Nov 01 09:57:23.777473 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.777451    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz" event={"ID":"8837ef1f-f84f-484d-89b4-49ef01386c8c","Type":"ContainerDied","Data":"d695316720084ee2db956091ffba4a7d7500677bd6b6676510e85a99d35f64c0"}
Nov 01 09:57:23.777609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.777457    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"
Nov 01 09:57:23.788962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.788892    4314 scope.go:122] "RemoveContainer" containerID="c08af440eaa276c7f542d9b0825c06b6fcef4d1e31528122d9396dfe53471de2"
Nov 01 09:57:23.930295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.930265    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"]
Nov 01 09:57:23.935914 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.935887    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"]
Nov 01 09:57:23.936114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.936093    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" containerID="cri-o://fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a" gracePeriod=30
Nov 01 09:57:23.978477 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:23.978449    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-tt7l8"]
Nov 01 09:57:24.081922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:24.081834    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"]
Nov 01 09:57:24.084651 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:24.084626    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4x6zz"]
Nov 01 09:57:24.928044 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:24.928013    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" path="/var/lib/kubelet/pods/2122f13b-f67a-43bc-abcf-7e434a8b2eff/volumes"
Nov 01 09:57:24.928565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:24.928549    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" path="/var/lib/kubelet/pods/32b83241-0e9e-4599-83c6-77ac7521427e/volumes"
Nov 01 09:57:24.929015 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:24.929002    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" path="/var/lib/kubelet/pods/8837ef1f-f84f-484d-89b4-49ef01386c8c/volumes"
Nov 01 09:57:24.929742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:24.929728    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" path="/var/lib/kubelet/pods/b485d1df-e0e6-46a9-87b2-6aa46a71c488/volumes"
Nov 01 09:57:26.968213 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:26.968165    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:28.681673 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:28.681639    4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:57:28.682085 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:28.681701    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:57:29.199317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.199256    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:57:29.288251 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.288220    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:29.310510 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310476    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310516    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310598    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310653    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310688    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310726    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310767    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.310947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310793    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:29.310947 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.310867    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.311167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311013    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:29.311167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311026    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data\") pod \"21560010-c52f-468f-b441-9a3f8ac3e2fd\" (UID: \"21560010-c52f-468f-b441-9a3f8ac3e2fd\") "
Nov 01 09:57:29.311167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311069    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:29.311167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311149    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:29.311402 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311378    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:29.311632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311611    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.311723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311635    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.311723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311649    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21560010-c52f-468f-b441-9a3f8ac3e2fd-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.311723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311662    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.311723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.311679    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.312977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.312949    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:57:29.313072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.312991    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "kube-api-access-hgfcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:29.313189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.313170    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:29.313189 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.313184    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy" pod "21560010-c52f-468f-b441-9a3f8ac3e2fd" (UID: "21560010-c52f-468f-b441-9a3f8ac3e2fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:29.413356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.413322    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.413356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.413351    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.413356 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.413365    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21560010-c52f-468f-b441-9a3f8ac3e2fd-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.413637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.413375    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfcj\" (UniqueName: \"kubernetes.io/projected/21560010-c52f-468f-b441-9a3f8ac3e2fd-kube-api-access-hgfcj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:29.816043 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.816010    4314 generic.go:356] "Generic (PLEG): container finished" podID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerID="fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a" exitCode=0
Nov 01 09:57:29.816507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.816060    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"
Nov 01 09:57:29.816507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.816095    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" event={"ID":"21560010-c52f-468f-b441-9a3f8ac3e2fd","Type":"ContainerDied","Data":"fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a"}
Nov 01 09:57:29.816507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.816152    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv" event={"ID":"21560010-c52f-468f-b441-9a3f8ac3e2fd","Type":"ContainerDied","Data":"76c9a31825f999d229b594133cdaf4835c5f59768d5919a06ac6d5d7d90aadb8"}
Nov 01 09:57:29.816507 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.816177    4314 scope.go:122] "RemoveContainer" containerID="fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a"
Nov 01 09:57:29.826743 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.826721    4314 scope.go:122] "RemoveContainer" containerID="fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a"
Nov 01 09:57:29.827064 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:29.827042    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a\": container with ID starting with fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a not found: ID does not exist" containerID="fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a"
Nov 01 09:57:29.827133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.827076    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a"} err="failed to get container status \"fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a\": rpc error: code = NotFound desc = could not find container \"fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a\": container with ID starting with fef7104dfc1a350193bae342b9d5b8098799979c495ec040494c050fb4f5149a not found: ID does not exist"
Nov 01 09:57:29.970639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:29.970606    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"]
Nov 01 09:57:30.016821 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:30.016788    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-5cvbv"]
Nov 01 09:57:30.940441 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:30.940401    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" path="/var/lib/kubelet/pods/21560010-c52f-468f-b441-9a3f8ac3e2fd/volumes"
Nov 01 09:57:31.736938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:31.736898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:36.583337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:36.583305    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:41.322587 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:57:41.322503    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:57:41.461338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:41.461304    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:45.153425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.153376    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"]
Nov 01 09:57:45.154410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154383    4314 cpu_manager.go:414] "containerMap: removing container" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy"
Nov 01 09:57:45.154410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154410    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154473    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerName="payload-processing"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154483    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerName="payload-processing"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154513    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d56ad473-a2a7-4da5-87ed-1929a0ee715b" containerName="payload-pre-processing"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154521    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56ad473-a2a7-4da5-87ed-1929a0ee715b" containerName="payload-pre-processing"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154572    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerName="payload-pre-processing"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154586    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerName="payload-pre-processing"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154601    4314 cpu_manager.go:414] "containerMap: removing container" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154609    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy"
Nov 01 09:57:45.154634 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154632    4314 cpu_manager.go:414] "containerMap: removing container" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154640    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154864    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8837ef1f-f84f-484d-89b4-49ef01386c8c" containerName="istio-proxy"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154882    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b485d1df-e0e6-46a9-87b2-6aa46a71c488" containerName="payload-processing"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154903    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56ad473-a2a7-4da5-87ed-1929a0ee715b" containerName="payload-pre-processing"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154913    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2122f13b-f67a-43bc-abcf-7e434a8b2eff" containerName="payload-pre-processing"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154928    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="21560010-c52f-468f-b441-9a3f8ac3e2fd" containerName="istio-proxy"
Nov 01 09:57:45.155129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.154945    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b83241-0e9e-4599-83c6-77ac7521427e" containerName="istio-proxy"
Nov 01 09:57:45.161142 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.161117    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.209973 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.209940    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"]
Nov 01 09:57:45.251841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.251809    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:57:45.251841 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.251810    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:57:45.252089 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.251880    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:57:45.255495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.255474    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:57:45.285155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.285127    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.285305 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.285280    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.285430 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.285408    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwq2\" (UniqueName: \"kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.386156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.386110    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.386386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.386178    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwq2\" (UniqueName: \"kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.386386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.386238    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.387094 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.387064    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.388609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.388583    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.444067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.443998    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwq2\" (UniqueName: \"kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2\") pod \"maas-api-revoke-sub-97cdac72-rw9sk\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.475963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.475931    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:45.844900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.844866    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"]
Nov 01 09:57:45.846597 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:57:45.846570    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00204c7c_c7b6_408f_b8e6_4426cecd2cee.slice/crio-af655a41f8c99d1dd47f05d91e10bd8191c26615c14790927519c67438590c0b WatchSource:0}: Error finding container af655a41f8c99d1dd47f05d91e10bd8191c26615c14790927519c67438590c0b: Status 404 returned error can't find the container with id af655a41f8c99d1dd47f05d91e10bd8191c26615c14790927519c67438590c0b
Nov 01 09:57:45.913594 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:45.913561    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk" event={"ID":"00204c7c-c7b6-408f-b8e6-4426cecd2cee","Type":"ContainerStarted","Data":"af655a41f8c99d1dd47f05d91e10bd8191c26615c14790927519c67438590c0b"}
Nov 01 09:57:46.294628 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:46.294571    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:46.921797 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:46.921758    4314 generic.go:356] "Generic (PLEG): container finished" podID="00204c7c-c7b6-408f-b8e6-4426cecd2cee" containerID="222d82a0278e5be4d94a975011a4afe5c9f902a0d1dce6616fe556ec911c6acf" exitCode=0
Nov 01 09:57:46.927949 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:46.927919    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk" event={"ID":"00204c7c-c7b6-408f-b8e6-4426cecd2cee","Type":"ContainerDied","Data":"222d82a0278e5be4d94a975011a4afe5c9f902a0d1dce6616fe556ec911c6acf"}
Nov 01 09:57:47.815744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.815714    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"]
Nov 01 09:57:47.819567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.819526    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.868381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.868351    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-fd5a10-openshift-default-dockercfg-r6v87\""
Nov 01 09:57:47.914895 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.914849    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915061 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.914905    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915078    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915134    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915267 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915251    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915318 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915303    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915369 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915340    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915449 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915424    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8c5\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.915498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.915475    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:47.917974 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:47.917951    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"]
Nov 01 09:57:48.017491 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017439    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017517    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017582    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017619    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8c5\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017694 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017649    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017697    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017722    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017798    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.017929 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.017852    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.018336 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.018311    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.018581 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.018552    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.018795 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.018777    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.019840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.019688    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.019840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.019795    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.021591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.021563    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.021591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.021584    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.076566 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.076467    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.076688 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.076582    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8c5\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.091781 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.091757    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:48.133071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.133042    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:48.220261 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.220232    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca\") pod \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") "
Nov 01 09:57:48.220439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.220383    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca\") pod \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") "
Nov 01 09:57:48.220439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.220425    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2\" (UniqueName: \"kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2\") pod \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\" (UID: \"00204c7c-c7b6-408f-b8e6-4426cecd2cee\") "
Nov 01 09:57:48.220848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.220822    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca" pod "00204c7c-c7b6-408f-b8e6-4426cecd2cee" (UID: "00204c7c-c7b6-408f-b8e6-4426cecd2cee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:48.222363 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.222339    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca" pod "00204c7c-c7b6-408f-b8e6-4426cecd2cee" (UID: "00204c7c-c7b6-408f-b8e6-4426cecd2cee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:57:48.222481 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.222461    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2" pod "00204c7c-c7b6-408f-b8e6-4426cecd2cee" (UID: "00204c7c-c7b6-408f-b8e6-4426cecd2cee"). InnerVolumeSpecName "kube-api-access-8rwq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:48.321965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.321922    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:48.321965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.321954    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00204c7c-c7b6-408f-b8e6-4426cecd2cee-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:48.321965 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.321965    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwq2\" (UniqueName: \"kubernetes.io/projected/00204c7c-c7b6-408f-b8e6-4426cecd2cee-kube-api-access-8rwq2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:48.550067 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.550038    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"]
Nov 01 09:57:48.550733 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:57:48.550710    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d489586_e677_4977_a073_3cd2cc196e55.slice/crio-d489a5ff2056b0f63c3564de0cf55a68e35667503ac5b80cce7f2b85ff99b4cc WatchSource:0}: Error finding container d489a5ff2056b0f63c3564de0cf55a68e35667503ac5b80cce7f2b85ff99b4cc: Status 404 returned error can't find the container with id d489a5ff2056b0f63c3564de0cf55a68e35667503ac5b80cce7f2b85ff99b4cc
Nov 01 09:57:48.553129 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.553094    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:57:48.553234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.553178    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:57:48.553234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.553227    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 09:57:48.937346 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.937289    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" event={"ID":"0d489586-e677-4977-a073-3cd2cc196e55","Type":"ContainerStarted","Data":"512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f"}
Nov 01 09:57:48.937739 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.937352    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" event={"ID":"0d489586-e677-4977-a073-3cd2cc196e55","Type":"ContainerStarted","Data":"d489a5ff2056b0f63c3564de0cf55a68e35667503ac5b80cce7f2b85ff99b4cc"}
Nov 01 09:57:48.939097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.939070    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk" event={"ID":"00204c7c-c7b6-408f-b8e6-4426cecd2cee","Type":"ContainerDied","Data":"af655a41f8c99d1dd47f05d91e10bd8191c26615c14790927519c67438590c0b"}
Nov 01 09:57:48.939223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.939123    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af655a41f8c99d1dd47f05d91e10bd8191c26615c14790927519c67438590c0b"
Nov 01 09:57:48.939223 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:48.939078    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"
Nov 01 09:57:49.053025 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:49.052975    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podStartSLOduration=2.052962185 podStartE2EDuration="2.052962185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:57:49.050524967 +0000 UTC m=+5142.244919533" watchObservedRunningTime="2026-11-01 09:57:49.052962185 +0000 UTC m=+5142.247356737"
Nov 01 09:57:49.133900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:49.133870    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:49.135444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:49.135417    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Nov 01 09:57:49.135618 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:49.135471    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Nov 01 09:57:50.134146 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:50.134095    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Nov 01 09:57:50.134668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:50.134167    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Nov 01 09:57:50.449839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:50.449765    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:51.133981 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:51.133945    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Nov 01 09:57:51.134175 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:51.133996    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Nov 01 09:57:52.134312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:52.134279    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Nov 01 09:57:52.134744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:52.134332    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Nov 01 09:57:52.213752 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:52.213713    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:53.133957 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:53.133924    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Nov 01 09:57:53.134139 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:53.133974    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Nov 01 09:57:54.133989 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:54.133956    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Nov 01 09:57:54.134387 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:54.134007    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Nov 01 09:57:55.138217 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.138186    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:55.138742 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.138626    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:55.139310 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.139290    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:57:55.878239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.878203    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"]
Nov 01 09:57:55.878732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.878718    4314 cpu_manager.go:414] "containerMap: removing container" podUID="00204c7c-c7b6-408f-b8e6-4426cecd2cee" containerName="revoke-keys"
Nov 01 09:57:55.878798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.878733    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00204c7c-c7b6-408f-b8e6-4426cecd2cee" containerName="revoke-keys"
Nov 01 09:57:55.878839 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.878820    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="00204c7c-c7b6-408f-b8e6-4426cecd2cee" containerName="revoke-keys"
Nov 01 09:57:55.882097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.882080    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:55.933386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.933357    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:57:55.933518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.933425    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:57:55.933518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.933429    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:57:55.933518 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.933493    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:57:55.937119 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:55.937092    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"]
Nov 01 09:57:56.034865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.034824    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.034865 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.034870    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.035101 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.035025    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2tf\" (UniqueName: \"kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.135994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.135916    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.135994 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.135950    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.136206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.136180    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2tf\" (UniqueName: \"kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.136636 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.136614    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.138395 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.138378    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.194035 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.193994    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2tf\" (UniqueName: \"kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf\") pod \"maas-api-revoke-sub-13753ffb-nf988\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.195813 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.195794    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:56.566278 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:56.566247    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"]
Nov 01 09:57:56.568168 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:57:56.568139    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192ec866_1ee6_437a_a406_ea4e5b73deae.slice/crio-38bfd3ba8306eeedaf9de1c68ca9f8d14df891bb0fa7f911eada62489205a195 WatchSource:0}: Error finding container 38bfd3ba8306eeedaf9de1c68ca9f8d14df891bb0fa7f911eada62489205a195: Status 404 returned error can't find the container with id 38bfd3ba8306eeedaf9de1c68ca9f8d14df891bb0fa7f911eada62489205a195
Nov 01 09:57:57.015431 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:57.015394    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988" event={"ID":"192ec866-1ee6-437a-a406-ea4e5b73deae","Type":"ContainerStarted","Data":"0884fd5d7f53f320625d17de919ea710179965339ad1db947f98fc5e2f2f1b34"}
Nov 01 09:57:57.015637 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:57.015452    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988" event={"ID":"192ec866-1ee6-437a-a406-ea4e5b73deae","Type":"ContainerStarted","Data":"38bfd3ba8306eeedaf9de1c68ca9f8d14df891bb0fa7f911eada62489205a195"}
Nov 01 09:57:57.072484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:57.072451    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:57:57.125511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:57.125405    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988" podStartSLOduration=2.125390693 podStartE2EDuration="2.125390693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:57:57.12339023 +0000 UTC m=+5150.317784785" watchObservedRunningTime="2026-11-01 09:57:57.125390693 +0000 UTC m=+5150.319785248"
Nov 01 09:57:58.023012 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:58.022977    4314 generic.go:356] "Generic (PLEG): container finished" podID="192ec866-1ee6-437a-a406-ea4e5b73deae" containerID="0884fd5d7f53f320625d17de919ea710179965339ad1db947f98fc5e2f2f1b34" exitCode=0
Nov 01 09:57:58.023406 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:58.023062    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988" event={"ID":"192ec866-1ee6-437a-a406-ea4e5b73deae","Type":"ContainerDied","Data":"0884fd5d7f53f320625d17de919ea710179965339ad1db947f98fc5e2f2f1b34"}
Nov 01 09:57:59.173616 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.173588    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:57:59.271166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.271133    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca\") pod \"192ec866-1ee6-437a-a406-ea4e5b73deae\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") "
Nov 01 09:57:59.271312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.271256    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf\" (UniqueName: \"kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf\") pod \"192ec866-1ee6-437a-a406-ea4e5b73deae\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") "
Nov 01 09:57:59.271312 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.271292    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca\") pod \"192ec866-1ee6-437a-a406-ea4e5b73deae\" (UID: \"192ec866-1ee6-437a-a406-ea4e5b73deae\") "
Nov 01 09:57:59.271749 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.271725    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca" pod "192ec866-1ee6-437a-a406-ea4e5b73deae" (UID: "192ec866-1ee6-437a-a406-ea4e5b73deae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:59.273364 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.273336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca" pod "192ec866-1ee6-437a-a406-ea4e5b73deae" (UID: "192ec866-1ee6-437a-a406-ea4e5b73deae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:57:59.273511 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.273494    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf" pod "192ec866-1ee6-437a-a406-ea4e5b73deae" (UID: "192ec866-1ee6-437a-a406-ea4e5b73deae"). InnerVolumeSpecName "kube-api-access-zh2tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:59.372863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.372828    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:59.372863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.372860    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2tf\" (UniqueName: \"kubernetes.io/projected/192ec866-1ee6-437a-a406-ea4e5b73deae-kube-api-access-zh2tf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:59.372863 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:57:59.372871    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ec866-1ee6-437a-a406-ea4e5b73deae-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:00.046389 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:00.046343    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988" event={"ID":"192ec866-1ee6-437a-a406-ea4e5b73deae","Type":"ContainerDied","Data":"38bfd3ba8306eeedaf9de1c68ca9f8d14df891bb0fa7f911eada62489205a195"}
Nov 01 09:58:00.046586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:00.046358    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"
Nov 01 09:58:00.046586 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:00.046465    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38bfd3ba8306eeedaf9de1c68ca9f8d14df891bb0fa7f911eada62489205a195"
Nov 01 09:58:00.630380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:00.630347    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:00.778453 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:00.778420    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:01.984736 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:01.984699    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:02.268622 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:02.268585    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:02.742690 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:02.742662    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"]
Nov 01 09:58:02.742891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:02.742871    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy" containerID="cri-o://c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3" gracePeriod=30
Nov 01 09:58:02.747097 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:02.747072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:02.880889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:02.880853    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"]
Nov 01 09:58:02.881156 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:02.881105    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy" containerID="cri-o://6803bb5a13ae3d628068d5f1d5f7b2d3f95e322d3a32c64767eda0c36a46cc74" gracePeriod=30
Nov 01 09:58:03.210883 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.210788    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:03.904334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.904298    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"]
Nov 01 09:58:03.908656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.906122    4314 cpu_manager.go:414] "containerMap: removing container" podUID="192ec866-1ee6-437a-a406-ea4e5b73deae" containerName="revoke-keys"
Nov 01 09:58:03.908656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.906149    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192ec866-1ee6-437a-a406-ea4e5b73deae" containerName="revoke-keys"
Nov 01 09:58:03.908656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.906378    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="192ec866-1ee6-437a-a406-ea4e5b73deae" containerName="revoke-keys"
Nov 01 09:58:03.911229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.911206    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:03.925418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.925396    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:03.925517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.925429    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfdq5\" (UniqueName: \"kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:03.925585 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.925560    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:03.962638 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.962608    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"]
Nov 01 09:58:03.967120 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.967082    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:58:03.967239 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:03.967148    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:58:04.004907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.004876    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:58:04.005075 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.005058    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:58:04.026141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.026116    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.026263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.026222    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.026263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.026246    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfdq5\" (UniqueName: \"kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.026960 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.026936    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.028574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.028525    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.083375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.083349    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfdq5\" (UniqueName: \"kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5\") pod \"maas-api-revoke-sub-b12a95ba-gqcnn\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.227182 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.227108    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:04.608962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:04.608934    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"]
Nov 01 09:58:04.610905 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:58:04.610880    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19fde78_6660_41c5_84f2_21675acca153.slice/crio-e3035e5df6c915495782e0c0a26dd8a6abce8b8ce52e8d7991ec3721b69b2a60 WatchSource:0}: Error finding container e3035e5df6c915495782e0c0a26dd8a6abce8b8ce52e8d7991ec3721b69b2a60: Status 404 returned error can't find the container with id e3035e5df6c915495782e0c0a26dd8a6abce8b8ce52e8d7991ec3721b69b2a60
Nov 01 09:58:05.077610 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:05.077569    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn" event={"ID":"a19fde78-6660-41c5-84f2-21675acca153","Type":"ContainerStarted","Data":"92748f11719a780ffe23dcaa42dc9351fc4b83def070ab4bed6c575e3079bab1"}
Nov 01 09:58:05.077758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:05.077636    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn" event={"ID":"a19fde78-6660-41c5-84f2-21675acca153","Type":"ContainerStarted","Data":"e3035e5df6c915495782e0c0a26dd8a6abce8b8ce52e8d7991ec3721b69b2a60"}
Nov 01 09:58:05.187582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:05.187444    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn" podStartSLOduration=2.187429333 podStartE2EDuration="2.187429333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:58:05.186028137 +0000 UTC m=+5158.380422691" watchObservedRunningTime="2026-11-01 09:58:05.187429333 +0000 UTC m=+5158.381823937"
Nov 01 09:58:06.084567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:06.084514    4314 generic.go:356] "Generic (PLEG): container finished" podID="a19fde78-6660-41c5-84f2-21675acca153" containerID="92748f11719a780ffe23dcaa42dc9351fc4b83def070ab4bed6c575e3079bab1" exitCode=0
Nov 01 09:58:06.084962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:06.084600    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn" event={"ID":"a19fde78-6660-41c5-84f2-21675acca153","Type":"ContainerDied","Data":"92748f11719a780ffe23dcaa42dc9351fc4b83def070ab4bed6c575e3079bab1"}
Nov 01 09:58:07.228332 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.228304    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:07.260457 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.260427    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca\") pod \"a19fde78-6660-41c5-84f2-21675acca153\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") "
Nov 01 09:58:07.260655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.260590    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5\" (UniqueName: \"kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5\") pod \"a19fde78-6660-41c5-84f2-21675acca153\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") "
Nov 01 09:58:07.260655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.260632    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca\") pod \"a19fde78-6660-41c5-84f2-21675acca153\" (UID: \"a19fde78-6660-41c5-84f2-21675acca153\") "
Nov 01 09:58:07.260825 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.260800    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca" pod "a19fde78-6660-41c5-84f2-21675acca153" (UID: "a19fde78-6660-41c5-84f2-21675acca153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:07.261069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.261050    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19fde78-6660-41c5-84f2-21675acca153-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:07.262614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.262586    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5" pod "a19fde78-6660-41c5-84f2-21675acca153" (UID: "a19fde78-6660-41c5-84f2-21675acca153"). InnerVolumeSpecName "kube-api-access-lfdq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:07.262732 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.262704    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca" pod "a19fde78-6660-41c5-84f2-21675acca153" (UID: "a19fde78-6660-41c5-84f2-21675acca153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:07.292263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.292234    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:07.362582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.362478    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfdq5\" (UniqueName: \"kubernetes.io/projected/a19fde78-6660-41c5-84f2-21675acca153-kube-api-access-lfdq5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:07.362582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.362505    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19fde78-6660-41c5-84f2-21675acca153-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:07.920564 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:07.918001    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:08.010080 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.010011    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:58:08.070357 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070325    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070632 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070606    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070674    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070751 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070706    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070749    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070784    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070825    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.070889 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070860    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.071109 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.070902    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket\") pod \"3ce27064-0747-46ee-ba8f-3733ebfde94b\" (UID: \"3ce27064-0747-46ee-ba8f-3733ebfde94b\") "
Nov 01 09:58:08.071655 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.071630    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.072468 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.072443    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.072676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.072652    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.072912 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.072847    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.074059 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.074017    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:08.074152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.074055    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:08.076911 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.076884    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "kube-api-access-mhc85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:08.077779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.077757    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.077857 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.077820    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo" pod "3ce27064-0747-46ee-ba8f-3733ebfde94b" (UID: "3ce27064-0747-46ee-ba8f-3733ebfde94b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:58:08.111789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.111750    4314 generic.go:356] "Generic (PLEG): container finished" podID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerID="6803bb5a13ae3d628068d5f1d5f7b2d3f95e322d3a32c64767eda0c36a46cc74" exitCode=0
Nov 01 09:58:08.111954 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.111818    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" event={"ID":"57c48ed1-1560-4392-ab4d-11622052f0f1","Type":"ContainerDied","Data":"6803bb5a13ae3d628068d5f1d5f7b2d3f95e322d3a32c64767eda0c36a46cc74"}
Nov 01 09:58:08.113706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.113685    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"
Nov 01 09:58:08.113927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.113698    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn" event={"ID":"a19fde78-6660-41c5-84f2-21675acca153","Type":"ContainerDied","Data":"e3035e5df6c915495782e0c0a26dd8a6abce8b8ce52e8d7991ec3721b69b2a60"}
Nov 01 09:58:08.114153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.114133    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3035e5df6c915495782e0c0a26dd8a6abce8b8ce52e8d7991ec3721b69b2a60"
Nov 01 09:58:08.115417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.115393    4314 generic.go:356] "Generic (PLEG): container finished" podID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerID="c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3" exitCode=0
Nov 01 09:58:08.115574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.115446    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"
Nov 01 09:58:08.115574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.115468    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" event={"ID":"3ce27064-0747-46ee-ba8f-3733ebfde94b","Type":"ContainerDied","Data":"c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3"}
Nov 01 09:58:08.115574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.115509    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd" event={"ID":"3ce27064-0747-46ee-ba8f-3733ebfde94b","Type":"ContainerDied","Data":"625da846cf491aa0bb05a1180da48949ce7db050bda2d96b3d68e62bf53ac393"}
Nov 01 09:58:08.115574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.115566    4314 scope.go:122] "RemoveContainer" containerID="c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3"
Nov 01 09:58:08.126835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.126431    4314 scope.go:122] "RemoveContainer" containerID="c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3"
Nov 01 09:58:08.126835 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:58:08.126761    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3\": container with ID starting with c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3 not found: ID does not exist" containerID="c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3"
Nov 01 09:58:08.126835 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.126802    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3"} err="failed to get container status \"c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3\": rpc error: code = NotFound desc = could not find container \"c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3\": container with ID starting with c29804722cb501c06aa30276646056ede5517625d12dfd8a98a57f2085045af3 not found: ID does not exist"
Nov 01 09:58:08.142678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.142657    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:58:08.172720 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172692    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.172891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172732    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.172891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172762    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.172891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172793    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.172891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172813    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.172891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172841    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.172891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172888    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.173345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.172943    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.173345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173028    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token\") pod \"57c48ed1-1560-4392-ab4d-11622052f0f1\" (UID: \"57c48ed1-1560-4392-ab4d-11622052f0f1\") "
Nov 01 09:58:08.173345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173176    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.173345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173187    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.173345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173275    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:08.173696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173506    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.173696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173562    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.173696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173652    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c48ed1-1560-4392-ab4d-11622052f0f1-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173673    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173690    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173705    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173719    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173732    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173746    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173760    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173775    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173789    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce27064-0747-46ee-ba8f-3733ebfde94b-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173804    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173819    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce27064-0747-46ee-ba8f-3733ebfde94b-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.173992 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.173834    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhc85\" (UniqueName: \"kubernetes.io/projected/3ce27064-0747-46ee-ba8f-3733ebfde94b-kube-api-access-mhc85\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.175240 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.175203    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:58:08.175627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.175526    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "kube-api-access-lvxj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:08.175726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.175684    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:08.176083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.176058    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token" pod "57c48ed1-1560-4392-ab4d-11622052f0f1" (UID: "57c48ed1-1560-4392-ab4d-11622052f0f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:08.178726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.178704    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"]
Nov 01 09:58:08.274923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.274842    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvxj7\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-kube-api-access-lvxj7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.274923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.274881    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.274923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.274898    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.274923 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.274912    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c48ed1-1560-4392-ab4d-11622052f0f1-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.275391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.274927    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c48ed1-1560-4392-ab4d-11622052f0f1-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:08.285427 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.285383    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:08.450598 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.450560    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"]
Nov 01 09:58:08.455495 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.455467    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-9vmzd"]
Nov 01 09:58:08.928062 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:08.928033    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" path="/var/lib/kubelet/pods/3ce27064-0747-46ee-ba8f-3733ebfde94b/volumes"
Nov 01 09:58:09.123672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:09.123639    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"
Nov 01 09:58:09.123855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:09.123638    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj" event={"ID":"57c48ed1-1560-4392-ab4d-11622052f0f1","Type":"ContainerDied","Data":"cd06c45d5ebc9b8f6677e8c4bcdcde031966eabf9d7c87cf68af7b35cc0f6f14"}
Nov 01 09:58:09.123855 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:09.123775    4314 scope.go:122] "RemoveContainer" containerID="6803bb5a13ae3d628068d5f1d5f7b2d3f95e322d3a32c64767eda0c36a46cc74"
Nov 01 09:58:09.279486 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:09.279454    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"]
Nov 01 09:58:09.325727 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:09.325700    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-765gj"]
Nov 01 09:58:10.211875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:10.211842    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:10.927968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:10.927932    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" path="/var/lib/kubelet/pods/57c48ed1-1560-4392-ab4d-11622052f0f1/volumes"
Nov 01 09:58:11.322709 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:58:11.322672    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:58:12.209708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:12.209669    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:16.492962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:16.492921    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:16.943627 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:16.943528    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:17.209123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:17.209033    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:18.031411 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.031373    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:18.075717 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.075679    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"]
Nov 01 09:58:18.076232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076212    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a19fde78-6660-41c5-84f2-21675acca153" containerName="revoke-keys"
Nov 01 09:58:18.076359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076233    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19fde78-6660-41c5-84f2-21675acca153" containerName="revoke-keys"
Nov 01 09:58:18.076359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076248    4314 cpu_manager.go:414] "containerMap: removing container" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy"
Nov 01 09:58:18.076359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076255    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy"
Nov 01 09:58:18.076359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076314    4314 cpu_manager.go:414] "containerMap: removing container" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy"
Nov 01 09:58:18.076359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076323    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy"
Nov 01 09:58:18.076669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076446    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce27064-0747-46ee-ba8f-3733ebfde94b" containerName="istio-proxy"
Nov 01 09:58:18.076669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076462    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19fde78-6660-41c5-84f2-21675acca153" containerName="revoke-keys"
Nov 01 09:58:18.076669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.076473    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c48ed1-1560-4392-ab4d-11622052f0f1" containerName="istio-proxy"
Nov 01 09:58:18.081309 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.081289    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.136460 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.136429    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"]
Nov 01 09:58:18.178934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.178898    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:58:18.178934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.178939    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:58:18.179149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.178980    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:58:18.179149 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.179021    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:58:18.181830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.181805    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.181952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.181867    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.181997 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.181957    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2x9\" (UniqueName: \"kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.283064 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.282987    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.283215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.283063    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.283215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.283122    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2x9\" (UniqueName: \"kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.283715 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.283688    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.285410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.285387    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.343958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.343933    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2x9\" (UniqueName: \"kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9\") pod \"maas-api-revoke-sub-19beba3f-n9p46\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.395315 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.395271    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:18.635355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.635265    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:18.816190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.816147    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"]
Nov 01 09:58:18.842267 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:58:18.842239    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49bdc99_fdc6_40c6_8027_43d0616652b5.slice/crio-6ae4c0aff7944850a9074d48fa04ad4874ea6594c843386b8ca21f299e2f37ce WatchSource:0}: Error finding container 6ae4c0aff7944850a9074d48fa04ad4874ea6594c843386b8ca21f299e2f37ce: Status 404 returned error can't find the container with id 6ae4c0aff7944850a9074d48fa04ad4874ea6594c843386b8ca21f299e2f37ce
Nov 01 09:58:18.899281 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:18.899224    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:19.187845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:19.187751    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46" event={"ID":"a49bdc99-fdc6-40c6-8027-43d0616652b5","Type":"ContainerStarted","Data":"2217efe939abe6218b4111944a69e044e91423d43637ce3e0b9af003c5b15f27"}
Nov 01 09:58:19.187845 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:19.187821    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46" event={"ID":"a49bdc99-fdc6-40c6-8027-43d0616652b5","Type":"ContainerStarted","Data":"6ae4c0aff7944850a9074d48fa04ad4874ea6594c843386b8ca21f299e2f37ce"}
Nov 01 09:58:19.300180 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:19.300130    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46" podStartSLOduration=1.300116067 podStartE2EDuration="1.300116067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:58:19.299195912 +0000 UTC m=+5172.493590468" watchObservedRunningTime="2026-11-01 09:58:19.300116067 +0000 UTC m=+5172.494510621"
Nov 01 09:58:20.195484 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:20.195449    4314 generic.go:356] "Generic (PLEG): container finished" podID="a49bdc99-fdc6-40c6-8027-43d0616652b5" containerID="2217efe939abe6218b4111944a69e044e91423d43637ce3e0b9af003c5b15f27" exitCode=0
Nov 01 09:58:20.195936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:20.195565    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46" event={"ID":"a49bdc99-fdc6-40c6-8027-43d0616652b5","Type":"ContainerDied","Data":"2217efe939abe6218b4111944a69e044e91423d43637ce3e0b9af003c5b15f27"}
Nov 01 09:58:21.336209 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.336185    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:21.419642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.419614    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9\" (UniqueName: \"kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9\") pod \"a49bdc99-fdc6-40c6-8027-43d0616652b5\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") "
Nov 01 09:58:21.419808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.419775    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca\") pod \"a49bdc99-fdc6-40c6-8027-43d0616652b5\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") "
Nov 01 09:58:21.419808 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.419803    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca\") pod \"a49bdc99-fdc6-40c6-8027-43d0616652b5\" (UID: \"a49bdc99-fdc6-40c6-8027-43d0616652b5\") "
Nov 01 09:58:21.420202 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.420173    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca" pod "a49bdc99-fdc6-40c6-8027-43d0616652b5" (UID: "a49bdc99-fdc6-40c6-8027-43d0616652b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:21.421772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.421750    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9" pod "a49bdc99-fdc6-40c6-8027-43d0616652b5" (UID: "a49bdc99-fdc6-40c6-8027-43d0616652b5"). InnerVolumeSpecName "kube-api-access-9g2x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:21.421842 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.421806    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca" pod "a49bdc99-fdc6-40c6-8027-43d0616652b5" (UID: "a49bdc99-fdc6-40c6-8027-43d0616652b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:21.521420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.521388    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:21.521420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.521417    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49bdc99-fdc6-40c6-8027-43d0616652b5-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:21.521420 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.521429    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2x9\" (UniqueName: \"kubernetes.io/projected/a49bdc99-fdc6-40c6-8027-43d0616652b5-kube-api-access-9g2x9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:21.981664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:21.981640    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:22.208100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:22.208066    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"
Nov 01 09:58:22.208100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:22.208084    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46" event={"ID":"a49bdc99-fdc6-40c6-8027-43d0616652b5","Type":"ContainerDied","Data":"6ae4c0aff7944850a9074d48fa04ad4874ea6594c843386b8ca21f299e2f37ce"}
Nov 01 09:58:22.208338 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:22.208136    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae4c0aff7944850a9074d48fa04ad4874ea6594c843386b8ca21f299e2f37ce"
Nov 01 09:58:22.485345 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:22.485302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:23.199206 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.199168    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" containerID="cri-o://aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461" gracePeriod=45
Nov 01 09:58:23.453187 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.453128    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:58:23.641508 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641466    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs\" (UniqueName: \"kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs\") pod \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") "
Nov 01 09:58:23.642017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641516    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm\") pod \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") "
Nov 01 09:58:23.642017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641607    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache\") pod \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") "
Nov 01 09:58:23.642017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641631    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd\" (UniqueName: \"kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd\") pod \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") "
Nov 01 09:58:23.642017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641766    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home\") pod \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") "
Nov 01 09:58:23.642017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641839    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir\") pod \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\" (UID: \"369d8ecd-2471-4aab-a1c3-e099e2fb5dda\") "
Nov 01 09:58:23.642323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.642088    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home" pod "369d8ecd-2471-4aab-a1c3-e099e2fb5dda" (UID: "369d8ecd-2471-4aab-a1c3-e099e2fb5dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.642323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.642104    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir" pod "369d8ecd-2471-4aab-a1c3-e099e2fb5dda" (UID: "369d8ecd-2471-4aab-a1c3-e099e2fb5dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.642323 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.641892    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache" pod "369d8ecd-2471-4aab-a1c3-e099e2fb5dda" (UID: "369d8ecd-2471-4aab-a1c3-e099e2fb5dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.642503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.642322    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.642503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.642367    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.642503 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.642384    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.643782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.643755    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm" pod "369d8ecd-2471-4aab-a1c3-e099e2fb5dda" (UID: "369d8ecd-2471-4aab-a1c3-e099e2fb5dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.643884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.643857    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs" pod "369d8ecd-2471-4aab-a1c3-e099e2fb5dda" (UID: "369d8ecd-2471-4aab-a1c3-e099e2fb5dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:23.643884 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.643873    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd" pod "369d8ecd-2471-4aab-a1c3-e099e2fb5dda" (UID: "369d8ecd-2471-4aab-a1c3-e099e2fb5dda"). InnerVolumeSpecName "kube-api-access-4j8qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:23.743428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.743397    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.743428 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.743425    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.743660 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:23.743437    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8qd\" (UniqueName: \"kubernetes.io/projected/369d8ecd-2471-4aab-a1c3-e099e2fb5dda-kube-api-access-4j8qd\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:24.221425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.221341    4314 generic.go:356] "Generic (PLEG): container finished" podID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerID="aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461" exitCode=0
Nov 01 09:58:24.221425 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.221409    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"
Nov 01 09:58:24.221702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.221424    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" event={"ID":"369d8ecd-2471-4aab-a1c3-e099e2fb5dda","Type":"ContainerDied","Data":"aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461"}
Nov 01 09:58:24.221702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.221470    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" event={"ID":"369d8ecd-2471-4aab-a1c3-e099e2fb5dda","Type":"ContainerDied","Data":"b60f7df135117c2b3147dca0dcd635f307db32e6be7102fe808c5bd68e333f7c"}
Nov 01 09:58:24.221702 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.221495    4314 scope.go:122] "RemoveContainer" containerID="aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461"
Nov 01 09:58:24.231045 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.231025    4314 scope.go:122] "RemoveContainer" containerID="aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461"
Nov 01 09:58:24.231311 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:58:24.231290    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461\": container with ID starting with aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461 not found: ID does not exist" containerID="aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461"
Nov 01 09:58:24.231381 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.231324    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461"} err="failed to get container status \"aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461\": rpc error: code = NotFound desc = could not find container \"aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461\": container with ID starting with aef125a4b389bedebc854387da5adcbcf307ad9470ff9f24ae755a6dd24d5461 not found: ID does not exist"
Nov 01 09:58:24.375678 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.375645    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"]
Nov 01 09:58:24.397830 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.397798    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:58:24.421800 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.421772    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/premium-w0-751a73-kserve-85668b8d94-79lbc"]
Nov 01 09:58:24.649779 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.649752    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:24.929292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:24.929215    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" path="/var/lib/kubelet/pods/369d8ecd-2471-4aab-a1c3-e099e2fb5dda/volumes"
Nov 01 09:58:27.283686 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:27.283655    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:27.790335 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:27.790303    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"]
Nov 01 09:58:27.998444 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:27.998408    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:28.176245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:28.176163    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:28.328000 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:28.327967    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:29.614292 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:29.614257    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:30.727232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:30.727199    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:31.898162 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:31.898130    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:32.791215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:32.791172    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerName="payload-pre-processing" containerID="cri-o://c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0" gracePeriod=25
Nov 01 09:58:33.859934 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:33.859890    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:34.205499 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:34.205407    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:36.858450 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:36.858418    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:37.008047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:37.007997    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:37.354959 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:37.354922    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:58:38.894398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:38.894364    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:41.323113 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:58:41.323080    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:58:43.680170 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:43.680141    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:44.820272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:44.820244    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:44.966434 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:44.966397    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:46.019758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:46.019723    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:47.354674 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:47.354637    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 09:58:48.598963 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:48.598931    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:49.351245 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:49.351213    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:49.852190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:49.852146    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:52.113205 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:52.113161    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:52.266455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:52.266422    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:53.045758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.045731    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:58:53.103493 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.103384    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:53.156472 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.156442    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5\" (UniqueName: \"kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5\") pod \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") "
Nov 01 09:58:53.157128 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.157101    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume\") pod \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\" (UID: \"0e7f8524-5bd8-412b-94e4-3a109a8a0170\") "
Nov 01 09:58:53.157591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.157513    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume" pod "0e7f8524-5bd8-412b-94e4-3a109a8a0170" (UID: "0e7f8524-5bd8-412b-94e4-3a109a8a0170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:53.157804 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.157782    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7f8524-5bd8-412b-94e4-3a109a8a0170-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:53.160975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.160947    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5" pod "0e7f8524-5bd8-412b-94e4-3a109a8a0170" (UID: "0e7f8524-5bd8-412b-94e4-3a109a8a0170"). InnerVolumeSpecName "kube-api-access-95mm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:53.258753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.258714    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mm5\" (UniqueName: \"kubernetes.io/projected/0e7f8524-5bd8-412b-94e4-3a109a8a0170-kube-api-access-95mm5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:53.384734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.384643    4314 generic.go:356] "Generic (PLEG): container finished" podID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerID="c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0" exitCode=0
Nov 01 09:58:53.384734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.384690    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"
Nov 01 09:58:53.384734 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.384710    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" event={"ID":"0e7f8524-5bd8-412b-94e4-3a109a8a0170","Type":"ContainerDied","Data":"c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0"}
Nov 01 09:58:53.385008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.384763    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk" event={"ID":"0e7f8524-5bd8-412b-94e4-3a109a8a0170","Type":"ContainerDied","Data":"79b2e0eef2220e96903df9b880aca1b39fdae42776e78895bc52ceb03426be06"}
Nov 01 09:58:53.385008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.384786    4314 scope.go:122] "RemoveContainer" containerID="c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0"
Nov 01 09:58:53.394656 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.394637    4314 scope.go:122] "RemoveContainer" containerID="c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0"
Nov 01 09:58:53.394886 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:58:53.394868    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0\": container with ID starting with c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0 not found: ID does not exist" containerID="c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0"
Nov 01 09:58:53.394933 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.394894    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0"} err="failed to get container status \"c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0\": rpc error: code = NotFound desc = could not find container \"c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0\": container with ID starting with c26e42270d7a8d7257a5dffc9a6b4cec0e17dea5db5d42e5b86c9996f4b30ef0 not found: ID does not exist"
Nov 01 09:58:53.538580 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.538551    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"]
Nov 01 09:58:53.545367 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:53.545340    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fd5a10-7755c8cf9f-v9vmk"]
Nov 01 09:58:54.927798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:54.927766    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" path="/var/lib/kubelet/pods/0e7f8524-5bd8-412b-94e4-3a109a8a0170/volumes"
Nov 01 09:58:54.984582 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:54.984526    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:58.597780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:58.597746    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:58.763271 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:58.763235    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:59.575076 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:59.575044    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:59.717995 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:59.717966    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:58:59.862680 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:58:59.862601    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:00.077024 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:00.076995    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"]
Nov 01 09:59:00.122583 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:00.122505    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d9e6d9-lctdk"]
Nov 01 09:59:00.927864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:00.927832    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ce3542-b285-4cb9-8684-01daae4f0efe" path="/var/lib/kubelet/pods/11ce3542-b285-4cb9-8684-01daae4f0efe/volumes"
Nov 01 09:59:01.194866 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:01.194784    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:01.682666 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:01.682636    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"]
Nov 01 09:59:01.682859 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:01.682835    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy" containerID="cri-o://512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f" gracePeriod=30
Nov 01 09:59:01.714416 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:01.714378    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:02.482069 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:02.482037    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:05.291892 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:05.291860    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:06.975307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.975280    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:59:06.995824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995804    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.995950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995853    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.995950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995879    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.995950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995903    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.995950 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995921    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.996181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995964    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.996181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.995991    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.996181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996117    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.996181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996139    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:06.996181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996160    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert\") pod \"0d489586-e677-4977-a073-3cd2cc196e55\" (UID: \"0d489586-e677-4977-a073-3cd2cc196e55\") "
Nov 01 09:59:06.996639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996564    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:06.996639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996579    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:06.996639 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996590    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:06.996913 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996764    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:06.996990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996909    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:06.996990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996934    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d489586-e677-4977-a073-3cd2cc196e55-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:06.996990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996950    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:06.996990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996964    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:06.996990 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.996979    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:06.998307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.998269    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:59:06.998307 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.998273    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:06.998463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.998405    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "kube-api-access-dm8c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:06.998463 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:06.998406    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy" pod "0d489586-e677-4977-a073-3cd2cc196e55" (UID: "0d489586-e677-4977-a073-3cd2cc196e55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:07.097952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.097925    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8c5\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-kube-api-access-dm8c5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:07.097952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.097948    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d489586-e677-4977-a073-3cd2cc196e55-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:07.097952 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.097958    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d489586-e677-4977-a073-3cd2cc196e55-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:07.098165 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.097968    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d489586-e677-4977-a073-3cd2cc196e55-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:07.466135 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.466107    4314 generic.go:356] "Generic (PLEG): container finished" podID="0d489586-e677-4977-a073-3cd2cc196e55" containerID="512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f" exitCode=0
Nov 01 09:59:07.466337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.466148    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"
Nov 01 09:59:07.466337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.466182    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" event={"ID":"0d489586-e677-4977-a073-3cd2cc196e55","Type":"ContainerDied","Data":"512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f"}
Nov 01 09:59:07.466337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.466244    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7" event={"ID":"0d489586-e677-4977-a073-3cd2cc196e55","Type":"ContainerDied","Data":"d489a5ff2056b0f63c3564de0cf55a68e35667503ac5b80cce7f2b85ff99b4cc"}
Nov 01 09:59:07.466337 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.466274    4314 scope.go:122] "RemoveContainer" containerID="512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f"
Nov 01 09:59:07.476502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.476486    4314 scope.go:122] "RemoveContainer" containerID="512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f"
Nov 01 09:59:07.476758 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:07.476741    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f\": container with ID starting with 512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f not found: ID does not exist" containerID="512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f"
Nov 01 09:59:07.476814 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.476767    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f"} err="failed to get container status \"512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f\": rpc error: code = NotFound desc = could not find container \"512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f\": container with ID starting with 512921203d67faf55f38892d8410249fbd83c37fa6b76df067d61e0a82b1886f not found: ID does not exist"
Nov 01 09:59:07.621317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.621281    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"]
Nov 01 09:59:07.625248 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:07.625230    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-nlgk7"]
Nov 01 09:59:08.008683 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:08.008632    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:08.156771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:08.156715    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:08.935263 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:08.935228    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d489586-e677-4977-a073-3cd2cc196e55" path="/var/lib/kubelet/pods/0d489586-e677-4977-a073-3cd2cc196e55/volumes"
Nov 01 09:59:09.091268 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:09.091224    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:10.417498 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:10.417464    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:11.321468 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:11.321433    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:59:12.452823 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:12.452793    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:12.599417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:12.599345    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:14.765801 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:14.765767    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:15.371840 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:15.371802    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:17.045163 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:17.045127    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"]
Nov 01 09:59:17.091114 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:17.091085    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bf608e-bj2sn"]
Nov 01 09:59:18.927975 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:18.927939    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080dd537-a63d-4de0-b4a2-2c1406e7005c" path="/var/lib/kubelet/pods/080dd537-a63d-4de0-b4a2-2c1406e7005c/volumes"
Nov 01 09:59:20.214570 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:20.214474    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:21.704765 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:21.704722    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:21.847870 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:21.847836    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:24.357467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:24.357440    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:26.183155 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:26.183102    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:29.965716 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:29.965684    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:30.205429 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:30.205393    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:30.477153 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:30.477115    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:31.074084 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:31.074042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:31.818198 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:31.818165    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:32.355574 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:32.355528    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"]
Nov 01 09:59:32.356047 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:32.355804    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy" containerID="cri-o://7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4" gracePeriod=30
Nov 01 09:59:32.363877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:32.363852    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:32.828088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:32.828045    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:33.421958 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.421925    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"]
Nov 01 09:59:33.422375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422351    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy"
Nov 01 09:59:33.422375 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422362    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy"
Nov 01 09:59:33.422462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422378    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a49bdc99-fdc6-40c6-8027-43d0616652b5" containerName="revoke-keys"
Nov 01 09:59:33.422462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422383    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49bdc99-fdc6-40c6-8027-43d0616652b5" containerName="revoke-keys"
Nov 01 09:59:33.422462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422403    4314 cpu_manager.go:414] "containerMap: removing container" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main"
Nov 01 09:59:33.422462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422408    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main"
Nov 01 09:59:33.422462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422415    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerName="payload-pre-processing"
Nov 01 09:59:33.422462 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422420    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerName="payload-pre-processing"
Nov 01 09:59:33.422723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422480    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d489586-e677-4977-a073-3cd2cc196e55" containerName="istio-proxy"
Nov 01 09:59:33.422723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422489    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49bdc99-fdc6-40c6-8027-43d0616652b5" containerName="revoke-keys"
Nov 01 09:59:33.422723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422498    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="369d8ecd-2471-4aab-a1c3-e099e2fb5dda" containerName="main"
Nov 01 09:59:33.422723 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.422504    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7f8524-5bd8-412b-94e4-3a109a8a0170" containerName="payload-pre-processing"
Nov 01 09:59:33.425722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.425704    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.471215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.471185    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.471360 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.471240    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cgn\" (UniqueName: \"kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.471436 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.471366    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.475900 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.475868    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:59:33.479793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.479757    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"]
Nov 01 09:59:33.521864 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.521840    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:59:33.522008 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.521992    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:59:33.522077 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.522001    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:59:33.572509 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.572479    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cgn\" (UniqueName: \"kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.572682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.572559    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.572682 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.572629    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.573295 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.573275    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.574877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.574856    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.630027 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.630004    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cgn\" (UniqueName: \"kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn\") pod \"maas-api-revoke-sub-0230f1eb-dhr5z\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:33.738999 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:33.738975    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:34.111893 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:34.111868    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"]
Nov 01 09:59:34.113267 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:59:34.113240    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540ba017_f178_436c_b8b2_25987db2c84c.slice/crio-1c5a6b234baa4c57bc515b9cbea631f82f5133d4bea576de360e92618d0fd086 WatchSource:0}: Error finding container 1c5a6b234baa4c57bc515b9cbea631f82f5133d4bea576de360e92618d0fd086: Status 404 returned error can't find the container with id 1c5a6b234baa4c57bc515b9cbea631f82f5133d4bea576de360e92618d0fd086
Nov 01 09:59:34.621011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:34.620920    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z" event={"ID":"540ba017-f178-436c-b8b2-25987db2c84c","Type":"ContainerStarted","Data":"319e1aecd6fd8ed7d8fc896cf0115c35271815e221094c1726a626b6f9abda3b"}
Nov 01 09:59:34.621011 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:34.620980    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z" event={"ID":"540ba017-f178-436c-b8b2-25987db2c84c","Type":"ContainerStarted","Data":"1c5a6b234baa4c57bc515b9cbea631f82f5133d4bea576de360e92618d0fd086"}
Nov 01 09:59:34.734439 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:34.734385    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z" podStartSLOduration=1.734365239 podStartE2EDuration="1.734365239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:59:34.731441098 +0000 UTC m=+5247.925835652" watchObservedRunningTime="2026-11-01 09:59:34.734365239 +0000 UTC m=+5247.928759795"
Nov 01 09:59:35.628200 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:35.628167    4314 generic.go:356] "Generic (PLEG): container finished" podID="540ba017-f178-436c-b8b2-25987db2c84c" containerID="319e1aecd6fd8ed7d8fc896cf0115c35271815e221094c1726a626b6f9abda3b" exitCode=0
Nov 01 09:59:35.628681 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:35.628248    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z" event={"ID":"540ba017-f178-436c-b8b2-25987db2c84c","Type":"ContainerDied","Data":"319e1aecd6fd8ed7d8fc896cf0115c35271815e221094c1726a626b6f9abda3b"}
Nov 01 09:59:36.375836 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.375803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:36.763002 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.762979    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:36.805299 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.805271    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn\" (UniqueName: \"kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn\") pod \"540ba017-f178-436c-b8b2-25987db2c84c\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") "
Nov 01 09:59:36.805442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.805314    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca\") pod \"540ba017-f178-436c-b8b2-25987db2c84c\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") "
Nov 01 09:59:36.805442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.805352    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca\") pod \"540ba017-f178-436c-b8b2-25987db2c84c\" (UID: \"540ba017-f178-436c-b8b2-25987db2c84c\") "
Nov 01 09:59:36.805685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.805665    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca" pod "540ba017-f178-436c-b8b2-25987db2c84c" (UID: "540ba017-f178-436c-b8b2-25987db2c84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:36.805815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.805804    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:36.807316 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.807295    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca" pod "540ba017-f178-436c-b8b2-25987db2c84c" (UID: "540ba017-f178-436c-b8b2-25987db2c84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:36.807384 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.807362    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn" pod "540ba017-f178-436c-b8b2-25987db2c84c" (UID: "540ba017-f178-436c-b8b2-25987db2c84c"). InnerVolumeSpecName "kube-api-access-k7cgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:36.906806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.906730    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cgn\" (UniqueName: \"kubernetes.io/projected/540ba017-f178-436c-b8b2-25987db2c84c-kube-api-access-k7cgn\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:36.906806 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:36.906767    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ba017-f178-436c-b8b2-25987db2c84c-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.229220 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.229130    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"]
Nov 01 09:59:37.229703 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.229683    4314 cpu_manager.go:414] "containerMap: removing container" podUID="540ba017-f178-436c-b8b2-25987db2c84c" containerName="revoke-keys"
Nov 01 09:59:37.229820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.229704    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540ba017-f178-436c-b8b2-25987db2c84c" containerName="revoke-keys"
Nov 01 09:59:37.229820 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.229803    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="540ba017-f178-436c-b8b2-25987db2c84c" containerName="revoke-keys"
Nov 01 09:59:37.233185 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.233165    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.310623 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.310589    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtjz\" (UniqueName: \"kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.310780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.310632    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.310780 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.310735    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.327088 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.327061    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"]
Nov 01 09:59:37.411400 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.411370    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtjz\" (UniqueName: \"kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.411569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.411415    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.411569 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.411451    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.412124 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.412101    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.413696 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.413674    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.468962 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.468931    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtjz\" (UniqueName: \"kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz\") pod \"maas-api-revoke-sub-938ca959-9twpr\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.548023 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.547993    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:37.601728 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.601688    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:59:37.613897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.613827    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.613897 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.613886    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.613938    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.613969    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.614034    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614090 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.614068    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.614160    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.614215    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.614391 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.614285    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert\") pod \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\" (UID: \"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1\") "
Nov 01 09:59:37.615606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.615061    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:37.615606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.615155    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:37.615606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.615357    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:37.615606 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.615506    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:37.616017 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.615972    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:37.619454 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.619432    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "kube-api-access-b7t5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:37.620614 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.620588    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:37.620693 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.620604    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:59:37.621824 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.621805    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy" pod "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" (UID: "823fb4f0-2d4a-4e2d-ac37-3c03749efbf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:37.644630 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.644606    4314 generic.go:356] "Generic (PLEG): container finished" podID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerID="7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4" exitCode=0
Nov 01 09:59:37.644767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.644664    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"
Nov 01 09:59:37.644767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.644690    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" event={"ID":"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1","Type":"ContainerDied","Data":"7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4"}
Nov 01 09:59:37.644767 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.644741    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj" event={"ID":"823fb4f0-2d4a-4e2d-ac37-3c03749efbf1","Type":"ContainerDied","Data":"a5675fb01e5754539a75b38a64e89eff0b2eb4f58bd8e732cd6467a236f6666d"}
Nov 01 09:59:37.644940 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.644769    4314 scope.go:122] "RemoveContainer" containerID="7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4"
Nov 01 09:59:37.646815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.646367    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"
Nov 01 09:59:37.646815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.646391    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z" event={"ID":"540ba017-f178-436c-b8b2-25987db2c84c","Type":"ContainerDied","Data":"1c5a6b234baa4c57bc515b9cbea631f82f5133d4bea576de360e92618d0fd086"}
Nov 01 09:59:37.646815 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.646444    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5a6b234baa4c57bc515b9cbea631f82f5133d4bea576de360e92618d0fd086"
Nov 01 09:59:37.657726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.657710    4314 scope.go:122] "RemoveContainer" containerID="6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"
Nov 01 09:59:37.666122 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.666104    4314 scope.go:122] "RemoveContainer" containerID="7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4"
Nov 01 09:59:37.666389 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:37.666369    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4\": container with ID starting with 7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4 not found: ID does not exist" containerID="7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4"
Nov 01 09:59:37.666471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.666399    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4"} err="failed to get container status \"7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4\": rpc error: code = NotFound desc = could not find container \"7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4\": container with ID starting with 7545a99b205e5b379f0758c338f8f55ee629b00b47a654256f6eb83c6d5a61c4 not found: ID does not exist"
Nov 01 09:59:37.666471 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.666423    4314 scope.go:122] "RemoveContainer" containerID="6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"
Nov 01 09:59:37.666759 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:37.666739    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5\": container with ID starting with 6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5 not found: ID does not exist" containerID="6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"
Nov 01 09:59:37.666810 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.666767    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5"} err="failed to get container status \"6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5\": rpc error: code = NotFound desc = could not find container \"6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5\": container with ID starting with 6ec7bda601876bb0538ca2e3fa62af67091884caeb4771eeee4e42892f9e18a5 not found: ID does not exist"
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715511    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715563    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715579    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715593    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715605    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715619    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715632    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715645    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7t5h\" (UniqueName: \"kubernetes.io/projected/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-kube-api-access-b7t5h\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.715817 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.715662    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:37.787906 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.787838    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"]
Nov 01 09:59:37.798232 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.798201    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"]
Nov 01 09:59:37.874385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.874350    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"]
Nov 01 09:59:37.878805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.878781    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-wrkkj"]
Nov 01 09:59:37.891757 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.891734    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:37.923601 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:37.923570    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"]
Nov 01 09:59:37.933307 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:59:37.933273    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29918d00_33ad_45c6_9885_430fe93efe2d.slice/crio-9e0091b7315c5de200c304be7877b19d942071a890c1e3ef24cd6700188e07b4 WatchSource:0}: Error finding container 9e0091b7315c5de200c304be7877b19d942071a890c1e3ef24cd6700188e07b4: Status 404 returned error can't find the container with id 9e0091b7315c5de200c304be7877b19d942071a890c1e3ef24cd6700188e07b4
Nov 01 09:59:38.654133 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:38.654092    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr" event={"ID":"29918d00-33ad-45c6-9885-430fe93efe2d","Type":"ContainerStarted","Data":"b85ef936ad95c57dbd119dd0d3bc4736c912ebe4f5df385c7fafdb362c501a49"}
Nov 01 09:59:38.654334 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:38.654157    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr" event={"ID":"29918d00-33ad-45c6-9885-430fe93efe2d","Type":"ContainerStarted","Data":"9e0091b7315c5de200c304be7877b19d942071a890c1e3ef24cd6700188e07b4"}
Nov 01 09:59:38.763873 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:38.763816    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr" podStartSLOduration=1.763802335 podStartE2EDuration="1.763802335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:59:38.761492441 +0000 UTC m=+5251.955887002" watchObservedRunningTime="2026-11-01 09:59:38.763802335 +0000 UTC m=+5251.958196888"
Nov 01 09:59:38.928607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:38.928528    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" path="/var/lib/kubelet/pods/823fb4f0-2d4a-4e2d-ac37-3c03749efbf1/volumes"
Nov 01 09:59:39.045152 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.045122    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"]
Nov 01 09:59:39.049036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.049010    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17191c34-tpqmz"]
Nov 01 09:59:39.377470 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.377437    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:39.543792 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.543758    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:39.661644 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.661571    4314 generic.go:356] "Generic (PLEG): container finished" podID="29918d00-33ad-45c6-9885-430fe93efe2d" containerID="b85ef936ad95c57dbd119dd0d3bc4736c912ebe4f5df385c7fafdb362c501a49" exitCode=0
Nov 01 09:59:39.661793 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.661658    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr" event={"ID":"29918d00-33ad-45c6-9885-430fe93efe2d","Type":"ContainerDied","Data":"b85ef936ad95c57dbd119dd0d3bc4736c912ebe4f5df385c7fafdb362c501a49"}
Nov 01 09:59:39.787121 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:39.787064    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:40.800418 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.800396    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:40.845607 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.845576    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca\") pod \"29918d00-33ad-45c6-9885-430fe93efe2d\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") "
Nov 01 09:59:40.845744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.845634    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz\" (UniqueName: \"kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz\") pod \"29918d00-33ad-45c6-9885-430fe93efe2d\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") "
Nov 01 09:59:40.845744 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.845722    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca\") pod \"29918d00-33ad-45c6-9885-430fe93efe2d\" (UID: \"29918d00-33ad-45c6-9885-430fe93efe2d\") "
Nov 01 09:59:40.845968 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.845942    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca" pod "29918d00-33ad-45c6-9885-430fe93efe2d" (UID: "29918d00-33ad-45c6-9885-430fe93efe2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:40.847664 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.847636    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca" pod "29918d00-33ad-45c6-9885-430fe93efe2d" (UID: "29918d00-33ad-45c6-9885-430fe93efe2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:40.847789 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.847750    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz" pod "29918d00-33ad-45c6-9885-430fe93efe2d" (UID: "29918d00-33ad-45c6-9885-430fe93efe2d"). InnerVolumeSpecName "kube-api-access-qgtjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:40.928711 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.928630    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9805952-12b5-449d-a33c-bc3b089185e3" path="/var/lib/kubelet/pods/b9805952-12b5-449d-a33c-bc3b089185e3/volumes"
Nov 01 09:59:40.946668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.946640    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:40.946668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.946662    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgtjz\" (UniqueName: \"kubernetes.io/projected/29918d00-33ad-45c6-9885-430fe93efe2d-kube-api-access-qgtjz\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:40.946668 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:40.946673    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29918d00-33ad-45c6-9885-430fe93efe2d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:41.196848 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:41.196770    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:41.321628 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:41.321587    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 09:59:41.674311 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:41.674279    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr" event={"ID":"29918d00-33ad-45c6-9885-430fe93efe2d","Type":"ContainerDied","Data":"9e0091b7315c5de200c304be7877b19d942071a890c1e3ef24cd6700188e07b4"}
Nov 01 09:59:41.674642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:41.674327    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e0091b7315c5de200c304be7877b19d942071a890c1e3ef24cd6700188e07b4"
Nov 01 09:59:41.674642 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:41.674292    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"
Nov 01 09:59:45.215669 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.215634    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"]
Nov 01 09:59:45.216072 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216059    4314 cpu_manager.go:414] "containerMap: removing container" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy"
Nov 01 09:59:45.216116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216072    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy"
Nov 01 09:59:45.216116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216085    4314 cpu_manager.go:414] "containerMap: removing container" podUID="29918d00-33ad-45c6-9885-430fe93efe2d" containerName="revoke-keys"
Nov 01 09:59:45.216116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216090    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29918d00-33ad-45c6-9885-430fe93efe2d" containerName="revoke-keys"
Nov 01 09:59:45.216116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216098    4314 cpu_manager.go:414] "containerMap: removing container" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy"
Nov 01 09:59:45.216116 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216103    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy"
Nov 01 09:59:45.216280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216187    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy"
Nov 01 09:59:45.216280 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.216194    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="29918d00-33ad-45c6-9885-430fe93efe2d" containerName="revoke-keys"
Nov 01 09:59:45.219748 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.219724    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.225249 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.225226    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"]
Nov 01 09:59:45.270459 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.270433    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:59:45.270726 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.270623    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:59:45.270838 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.270816    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:59:45.271602 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.271578    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:59:45.274442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.274415    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"]
Nov 01 09:59:45.289976 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.289949    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.290096 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.290010    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8sc\" (UniqueName: \"kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.290421 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.290110    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.331772 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.331744    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"]
Nov 01 09:59:45.390758 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.390716    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.390936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.390768    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8sc\" (UniqueName: \"kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.390936 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.390794    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.391414 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.391395    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.393086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.393056    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.447899 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.447873    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8sc\" (UniqueName: \"kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc\") pod \"maas-api-revoke-sub-4c1fedfc-922lr\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.534386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.534361    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:45.635234 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.635200    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:45.786029 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.785945    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:45.901770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:45.901740    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"]
Nov 01 09:59:45.902917 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:59:45.902890    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbd8357_972c_4c64_a302_9519f7bdd93f.slice/crio-8dda1822bf13ae79133017c167182f77eea4ace6732e15a9851ce60ee439c065 WatchSource:0}: Error finding container 8dda1822bf13ae79133017c167182f77eea4ace6732e15a9851ce60ee439c065: Status 404 returned error can't find the container with id 8dda1822bf13ae79133017c167182f77eea4ace6732e15a9851ce60ee439c065
Nov 01 09:59:46.086086 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:46.086060    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:46.470287 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:46.470204    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:46.702770 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:46.702743    4314 generic.go:356] "Generic (PLEG): container finished" podID="9cbd8357-972c-4c64-a302-9519f7bdd93f" containerID="93a32d56c751d3a0d6a2ccf9420e75492e5c9723cba8cd0d5a8ed94c3a48258b" exitCode=0
Nov 01 09:59:46.702927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:46.702774    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr" event={"ID":"9cbd8357-972c-4c64-a302-9519f7bdd93f","Type":"ContainerDied","Data":"93a32d56c751d3a0d6a2ccf9420e75492e5c9723cba8cd0d5a8ed94c3a48258b"}
Nov 01 09:59:46.702927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:46.702825    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr" event={"ID":"9cbd8357-972c-4c64-a302-9519f7bdd93f","Type":"ContainerStarted","Data":"8dda1822bf13ae79133017c167182f77eea4ace6732e15a9851ce60ee439c065"}
Nov 01 09:59:47.840729 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.840708    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:47.912805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.912774    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc\" (UniqueName: \"kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc\") pod \"9cbd8357-972c-4c64-a302-9519f7bdd93f\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") "
Nov 01 09:59:47.912977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.912882    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca\") pod \"9cbd8357-972c-4c64-a302-9519f7bdd93f\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") "
Nov 01 09:59:47.912977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.912947    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca\") pod \"9cbd8357-972c-4c64-a302-9519f7bdd93f\" (UID: \"9cbd8357-972c-4c64-a302-9519f7bdd93f\") "
Nov 01 09:59:47.913237 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.913211    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca" pod "9cbd8357-972c-4c64-a302-9519f7bdd93f" (UID: "9cbd8357-972c-4c64-a302-9519f7bdd93f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:47.914860 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.914838    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc" pod "9cbd8357-972c-4c64-a302-9519f7bdd93f" (UID: "9cbd8357-972c-4c64-a302-9519f7bdd93f"). InnerVolumeSpecName "kube-api-access-8w8sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:47.914977 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.914953    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca" pod "9cbd8357-972c-4c64-a302-9519f7bdd93f" (UID: "9cbd8357-972c-4c64-a302-9519f7bdd93f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:47.938235 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.938212    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"]
Nov 01 09:59:47.938708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.938681    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9cbd8357-972c-4c64-a302-9519f7bdd93f" containerName="revoke-keys"
Nov 01 09:59:47.938708 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.938710    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbd8357-972c-4c64-a302-9519f7bdd93f" containerName="revoke-keys"
Nov 01 09:59:47.938877 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.938861    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="823fb4f0-2d4a-4e2d-ac37-3c03749efbf1" containerName="istio-proxy"
Nov 01 09:59:47.938938 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.938885    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbd8357-972c-4c64-a302-9519f7bdd93f" containerName="revoke-keys"
Nov 01 09:59:47.942126 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.942109    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:47.970100 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:47.970038    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"]
Nov 01 09:59:48.014380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.014353    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.014565 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.014504    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.014661 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.014612    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpb5\" (UniqueName: \"kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.014756 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.014739    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:48.014798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.014760    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbd8357-972c-4c64-a302-9519f7bdd93f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:48.014798 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.014770    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8sc\" (UniqueName: \"kubernetes.io/projected/9cbd8357-972c-4c64-a302-9519f7bdd93f-kube-api-access-8w8sc\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:48.115348 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.115317    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpb5\" (UniqueName: \"kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.115519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.115366    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.115519 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.115439    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.116051 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.116030    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.117722 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.117694    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.172676 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.172648    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpb5\" (UniqueName: \"kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5\") pod \"maas-api-revoke-sub-5887ebe1-9h4br\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.256123 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.256097    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:48.621517 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.621494    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"]
Nov 01 09:59:48.622671 ip-10-0-134-93 kubenswrapper[4314]: W1101 09:59:48.622646    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6599748_433a_4fad_81a1_a989d6df7271.slice/crio-e22461493bef379da12c24277cc54c65db75d6310e4a162f68661bd8afad273d WatchSource:0}: Error finding container e22461493bef379da12c24277cc54c65db75d6310e4a162f68661bd8afad273d: Status 404 returned error can't find the container with id e22461493bef379da12c24277cc54c65db75d6310e4a162f68661bd8afad273d
Nov 01 09:59:48.716492 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.716405    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br" event={"ID":"d6599748-433a-4fad-81a1-a989d6df7271","Type":"ContainerStarted","Data":"e22461493bef379da12c24277cc54c65db75d6310e4a162f68661bd8afad273d"}
Nov 01 09:59:48.718034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.717995    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr" event={"ID":"9cbd8357-972c-4c64-a302-9519f7bdd93f","Type":"ContainerDied","Data":"8dda1822bf13ae79133017c167182f77eea4ace6732e15a9851ce60ee439c065"}
Nov 01 09:59:48.718034 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.718021    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"
Nov 01 09:59:48.718215 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:48.718045    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dda1822bf13ae79133017c167182f77eea4ace6732e15a9851ce60ee439c065"
Nov 01 09:59:49.728811 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:49.728731    4314 generic.go:356] "Generic (PLEG): container finished" podID="d6599748-433a-4fad-81a1-a989d6df7271" containerID="ab057b57c70a42e1816fdb2be61082904a2c2e7ddb63d81721e3db72e0998294" exitCode=0
Nov 01 09:59:49.729184 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:49.728808    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br" event={"ID":"d6599748-433a-4fad-81a1-a989d6df7271","Type":"ContainerDied","Data":"ab057b57c70a42e1816fdb2be61082904a2c2e7ddb63d81721e3db72e0998294"}
Nov 01 09:59:50.045083 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.045049    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"]
Nov 01 09:59:50.049901 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.049867    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba313e68-dmhlm"]
Nov 01 09:59:50.225922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.225886    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg" podUID="d834dc53-e017-409a-8d6a-64960a1bc6f2" containerName="payload-processing" containerID="cri-o://074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e" gracePeriod=25
Nov 01 09:59:50.333103 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.332987    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr" podUID="dfb92746-a22a-4898-a45d-de0a9935931c" containerName="payload-pre-processing" containerID="cri-o://8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002" gracePeriod=25
Nov 01 09:59:50.477633 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.477607    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 09:59:50.539004 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.538973    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume\") pod \"d834dc53-e017-409a-8d6a-64960a1bc6f2\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") "
Nov 01 09:59:50.539166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.539092    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t\" (UniqueName: \"kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t\") pod \"d834dc53-e017-409a-8d6a-64960a1bc6f2\" (UID: \"d834dc53-e017-409a-8d6a-64960a1bc6f2\") "
Nov 01 09:59:50.539361 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.539335    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume" pod "d834dc53-e017-409a-8d6a-64960a1bc6f2" (UID: "d834dc53-e017-409a-8d6a-64960a1bc6f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:50.539521 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.539501    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834dc53-e017-409a-8d6a-64960a1bc6f2-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:50.541782 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.541756    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t" pod "d834dc53-e017-409a-8d6a-64960a1bc6f2" (UID: "d834dc53-e017-409a-8d6a-64960a1bc6f2"). InnerVolumeSpecName "kube-api-access-52p6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:50.568317 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.568298    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 09:59:50.641325 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.641245    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j\" (UniqueName: \"kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j\") pod \"dfb92746-a22a-4898-a45d-de0a9935931c\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") "
Nov 01 09:59:50.641475 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.641366    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume\") pod \"dfb92746-a22a-4898-a45d-de0a9935931c\" (UID: \"dfb92746-a22a-4898-a45d-de0a9935931c\") "
Nov 01 09:59:50.641685 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.641663    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52p6t\" (UniqueName: \"kubernetes.io/projected/d834dc53-e017-409a-8d6a-64960a1bc6f2-kube-api-access-52p6t\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:50.641805 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.641727    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume" pod "dfb92746-a22a-4898-a45d-de0a9935931c" (UID: "dfb92746-a22a-4898-a45d-de0a9935931c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:50.643417 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.643390    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j" pod "dfb92746-a22a-4898-a45d-de0a9935931c" (UID: "dfb92746-a22a-4898-a45d-de0a9935931c"). InnerVolumeSpecName "kube-api-access-jqh2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:50.736141 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.736109    4314 generic.go:356] "Generic (PLEG): container finished" podID="d834dc53-e017-409a-8d6a-64960a1bc6f2" containerID="074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e" exitCode=0
Nov 01 09:59:50.736621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.736161    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"
Nov 01 09:59:50.736621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.736188    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg" event={"ID":"d834dc53-e017-409a-8d6a-64960a1bc6f2","Type":"ContainerDied","Data":"074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e"}
Nov 01 09:59:50.736621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.736246    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg" event={"ID":"d834dc53-e017-409a-8d6a-64960a1bc6f2","Type":"ContainerDied","Data":"8131fd8f0c9f49a692da9e32ac2366fd8cc6964ec6d7cb1a060d6b4185b17731"}
Nov 01 09:59:50.736621 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.736273    4314 scope.go:122] "RemoveContainer" containerID="074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e"
Nov 01 09:59:50.737953 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.737923    4314 generic.go:356] "Generic (PLEG): container finished" podID="dfb92746-a22a-4898-a45d-de0a9935931c" containerID="8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002" exitCode=0
Nov 01 09:59:50.738082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.737976    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"
Nov 01 09:59:50.738082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.738006    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr" event={"ID":"dfb92746-a22a-4898-a45d-de0a9935931c","Type":"ContainerDied","Data":"8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002"}
Nov 01 09:59:50.738082 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.738058    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr" event={"ID":"dfb92746-a22a-4898-a45d-de0a9935931c","Type":"ContainerDied","Data":"c8f8b3e1d4f538c1c210f69fce38a81f78b73f599098b3248eddeb1a3a08eacb"}
Nov 01 09:59:50.742359 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.742328    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqh2j\" (UniqueName: \"kubernetes.io/projected/dfb92746-a22a-4898-a45d-de0a9935931c-kube-api-access-jqh2j\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:50.742455 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.742359    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb92746-a22a-4898-a45d-de0a9935931c-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:50.747922 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.747785    4314 scope.go:122] "RemoveContainer" containerID="074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e"
Nov 01 09:59:50.748258 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:50.748207    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e\": container with ID starting with 074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e not found: ID does not exist" containerID="074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e"
Nov 01 09:59:50.748258 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.748241    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e"} err="failed to get container status \"074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e\": rpc error: code = NotFound desc = could not find container \"074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e\": container with ID starting with 074bd21af0e0454cf302ae1b58840e3e4330afc74748f3fec95024bc8fc7213e not found: ID does not exist"
Nov 01 09:59:50.748410 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.748263    4314 scope.go:122] "RemoveContainer" containerID="8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002"
Nov 01 09:59:50.757125 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.757107    4314 scope.go:122] "RemoveContainer" containerID="8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002"
Nov 01 09:59:50.757415 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:50.757394    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002\": container with ID starting with 8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002 not found: ID does not exist" containerID="8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002"
Nov 01 09:59:50.757479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.757425    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002"} err="failed to get container status \"8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002\": rpc error: code = NotFound desc = could not find container \"8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002\": container with ID starting with 8622370c763bc0a4f7c1b1502975fa11929f369c1ef14150bcca8523d2460002 not found: ID does not exist"
Nov 01 09:59:50.857907 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.857885    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:50.891386 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.891306    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"]
Nov 01 09:59:50.895398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.895375    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b711ae-784465d969-mvttg"]
Nov 01 09:59:50.927867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.927842    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d74a94e-a3e7-4185-99d6-c3503a217f26" path="/var/lib/kubelet/pods/2d74a94e-a3e7-4185-99d6-c3503a217f26/volumes"
Nov 01 09:59:50.928272 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.928253    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d834dc53-e017-409a-8d6a-64960a1bc6f2" path="/var/lib/kubelet/pods/d834dc53-e017-409a-8d6a-64960a1bc6f2/volumes"
Nov 01 09:59:50.944448 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.944426    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca\") pod \"d6599748-433a-4fad-81a1-a989d6df7271\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") "
Nov 01 09:59:50.944572 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.944510    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5\" (UniqueName: \"kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5\") pod \"d6599748-433a-4fad-81a1-a989d6df7271\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") "
Nov 01 09:59:50.944663 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.944649    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca\") pod \"d6599748-433a-4fad-81a1-a989d6df7271\" (UID: \"d6599748-433a-4fad-81a1-a989d6df7271\") "
Nov 01 09:59:50.945071 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.945045    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca" pod "d6599748-433a-4fad-81a1-a989d6df7271" (UID: "d6599748-433a-4fad-81a1-a989d6df7271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:59:50.946591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.946566    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca" pod "d6599748-433a-4fad-81a1-a989d6df7271" (UID: "d6599748-433a-4fad-81a1-a989d6df7271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:50.946672 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.946623    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5" pod "d6599748-433a-4fad-81a1-a989d6df7271" (UID: "d6599748-433a-4fad-81a1-a989d6df7271"). InnerVolumeSpecName "kube-api-access-clpb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:50.996378 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:50.996353    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"]
Nov 01 09:59:51.000920 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.000897    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b711ae-8cb586f47-x6xtr"]
Nov 01 09:59:51.045591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.045547    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:51.045591 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.045587    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6599748-433a-4fad-81a1-a989d6df7271-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:51.045771 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.045601    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpb5\" (UniqueName: \"kubernetes.io/projected/d6599748-433a-4fad-81a1-a989d6df7271-kube-api-access-clpb5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:51.234854 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.234823    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:51.747404 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.747364    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br" event={"ID":"d6599748-433a-4fad-81a1-a989d6df7271","Type":"ContainerDied","Data":"e22461493bef379da12c24277cc54c65db75d6310e4a162f68661bd8afad273d"}
Nov 01 09:59:51.747819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.747419    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e22461493bef379da12c24277cc54c65db75d6310e4a162f68661bd8afad273d"
Nov 01 09:59:51.747819 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:51.747383    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"
Nov 01 09:59:52.318166 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:52.318135    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:52.807875 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:52.807843    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main" containerID="cri-o://ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e" gracePeriod=45
Nov 01 09:59:52.819941 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:52.819913    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main" containerID="cri-o://20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75" gracePeriod=45
Nov 01 09:59:52.927398 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:52.927370    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb92746-a22a-4898-a45d-de0a9935931c" path="/var/lib/kubelet/pods/dfb92746-a22a-4898-a45d-de0a9935931c/volumes"
Nov 01 09:59:53.085380 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.085356    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:59:53.088229 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.088208    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:59:53.167567 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167545    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs\" (UniqueName: \"kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs\") pod \"54afff9d-b553-41c2-8857-8a17fa342d12\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") "
Nov 01 09:59:53.167701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167586    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir\") pod \"54afff9d-b553-41c2-8857-8a17fa342d12\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") "
Nov 01 09:59:53.167701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167620    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home\") pod \"47c20553-56d3-442c-971c-4864a829112e\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") "
Nov 01 09:59:53.167701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167644    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr\" (UniqueName: \"kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr\") pod \"47c20553-56d3-442c-971c-4864a829112e\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") "
Nov 01 09:59:53.167701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167666    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir\") pod \"47c20553-56d3-442c-971c-4864a829112e\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") "
Nov 01 09:59:53.167701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167689    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm\") pod \"54afff9d-b553-41c2-8857-8a17fa342d12\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") "
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167709    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm\") pod \"47c20553-56d3-442c-971c-4864a829112e\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") "
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167745    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache\") pod \"47c20553-56d3-442c-971c-4864a829112e\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") "
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167779    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj\" (UniqueName: \"kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj\") pod \"54afff9d-b553-41c2-8857-8a17fa342d12\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") "
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167805    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs\" (UniqueName: \"kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs\") pod \"47c20553-56d3-442c-971c-4864a829112e\" (UID: \"47c20553-56d3-442c-971c-4864a829112e\") "
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167829    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir" pod "54afff9d-b553-41c2-8857-8a17fa342d12" (UID: "54afff9d-b553-41c2-8857-8a17fa342d12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167848    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache\") pod \"54afff9d-b553-41c2-8857-8a17fa342d12\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") "
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167895    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home" pod "47c20553-56d3-442c-971c-4864a829112e" (UID: "47c20553-56d3-442c-971c-4864a829112e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.167991 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.167932    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home\") pod \"54afff9d-b553-41c2-8857-8a17fa342d12\" (UID: \"54afff9d-b553-41c2-8857-8a17fa342d12\") "
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168042    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache" pod "47c20553-56d3-442c-971c-4864a829112e" (UID: "47c20553-56d3-442c-971c-4864a829112e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168161    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir" pod "47c20553-56d3-442c-971c-4864a829112e" (UID: "47c20553-56d3-442c-971c-4864a829112e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168395    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache" pod "54afff9d-b553-41c2-8857-8a17fa342d12" (UID: "54afff9d-b553-41c2-8857-8a17fa342d12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168416    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168439    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168452    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.168478 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168466    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.168917 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.168593    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home" pod "54afff9d-b553-41c2-8857-8a17fa342d12" (UID: "54afff9d-b553-41c2-8857-8a17fa342d12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.170158 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.170131    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj" pod "54afff9d-b553-41c2-8857-8a17fa342d12" (UID: "54afff9d-b553-41c2-8857-8a17fa342d12"). InnerVolumeSpecName "kube-api-access-9mhvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:53.170265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.170133    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm" pod "54afff9d-b553-41c2-8857-8a17fa342d12" (UID: "54afff9d-b553-41c2-8857-8a17fa342d12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.170265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.170220    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs" pod "54afff9d-b553-41c2-8857-8a17fa342d12" (UID: "54afff9d-b553-41c2-8857-8a17fa342d12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:53.170265 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.170238    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr" pod "47c20553-56d3-442c-971c-4864a829112e" (UID: "47c20553-56d3-442c-971c-4864a829112e"). InnerVolumeSpecName "kube-api-access-kkvxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:53.170510 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.170495    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs" pod "47c20553-56d3-442c-971c-4864a829112e" (UID: "47c20553-56d3-442c-971c-4864a829112e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:53.170609 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.170587    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm" pod "47c20553-56d3-442c-971c-4864a829112e" (UID: "47c20553-56d3-442c-971c-4864a829112e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.269190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269157    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269186    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269190 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269198    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54afff9d-b553-41c2-8857-8a17fa342d12-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269211    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvxr\" (UniqueName: \"kubernetes.io/projected/47c20553-56d3-442c-971c-4864a829112e-kube-api-access-kkvxr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269226    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54afff9d-b553-41c2-8857-8a17fa342d12-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269236    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c20553-56d3-442c-971c-4864a829112e-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269247    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhvj\" (UniqueName: \"kubernetes.io/projected/54afff9d-b553-41c2-8857-8a17fa342d12-kube-api-access-9mhvj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.269433 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.269258    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c20553-56d3-442c-971c-4864a829112e-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.760214 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.760174    4314 generic.go:356] "Generic (PLEG): container finished" podID="47c20553-56d3-442c-971c-4864a829112e" containerID="ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e" exitCode=0
Nov 01 09:59:53.760467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.760243    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"
Nov 01 09:59:53.760467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.760260    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" event={"ID":"47c20553-56d3-442c-971c-4864a829112e","Type":"ContainerDied","Data":"ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e"}
Nov 01 09:59:53.760467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.760313    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66" event={"ID":"47c20553-56d3-442c-971c-4864a829112e","Type":"ContainerDied","Data":"dd061363a2b0c0b6456dbdb12365d41949b37907d23098cb2396541e9bfe2738"}
Nov 01 09:59:53.760467 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.760337    4314 scope.go:122] "RemoveContainer" containerID="ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e"
Nov 01 09:59:53.761891 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.761864    4314 generic.go:356] "Generic (PLEG): container finished" podID="54afff9d-b553-41c2-8857-8a17fa342d12" containerID="20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75" exitCode=0
Nov 01 09:59:53.762036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.761950    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"
Nov 01 09:59:53.762036 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.761983    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" event={"ID":"54afff9d-b553-41c2-8857-8a17fa342d12","Type":"ContainerDied","Data":"20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75"}
Nov 01 09:59:53.762181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.762037    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj" event={"ID":"54afff9d-b553-41c2-8857-8a17fa342d12","Type":"ContainerDied","Data":"aee843a853f35d46c149c18fae85d3f037cebbffb691a7a03ebcf613d10b51d0"}
Nov 01 09:59:53.770927 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.770909    4314 scope.go:122] "RemoveContainer" containerID="ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e"
Nov 01 09:59:53.771246 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:53.771223    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e\": container with ID starting with ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e not found: ID does not exist" containerID="ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e"
Nov 01 09:59:53.771355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.771251    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e"} err="failed to get container status \"ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e\": rpc error: code = NotFound desc = could not find container \"ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e\": container with ID starting with ccdfa04e15c46dcb743b864af72b870dfe9ab1db084c6700f9faff7ba528f72e not found: ID does not exist"
Nov 01 09:59:53.771355 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.771274    4314 scope.go:122] "RemoveContainer" containerID="20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75"
Nov 01 09:59:53.780479 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.780461    4314 scope.go:122] "RemoveContainer" containerID="20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75"
Nov 01 09:59:53.780800 ip-10-0-134-93 kubenswrapper[4314]: E1101 09:59:53.780773    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75\": container with ID starting with 20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75 not found: ID does not exist" containerID="20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75"
Nov 01 09:59:53.780867 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.780808    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75"} err="failed to get container status \"20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75\": rpc error: code = NotFound desc = could not find container \"20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75\": container with ID starting with 20467ad0fae56ef1ee7b2d52ea96084f78a0f4bbbfd52b5d46e6e34490be5c75 not found: ID does not exist"
Nov 01 09:59:53.919714 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.919679    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"]
Nov 01 09:59:53.923706 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:53.923685    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/premium-w4-b711ae-kserve-f668d6bd7-tjptj"]
Nov 01 09:59:54.022392 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:54.022307    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"]
Nov 01 09:59:54.033277 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:54.033249    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-w4-b711ae-kserve-5db4856578-ftx66"]
Nov 01 09:59:54.928631 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:54.928590    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c20553-56d3-442c-971c-4864a829112e" path="/var/lib/kubelet/pods/47c20553-56d3-442c-971c-4864a829112e/volumes"
Nov 01 09:59:54.929167 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:54.929143    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" path="/var/lib/kubelet/pods/54afff9d-b553-41c2-8857-8a17fa342d12/volumes"
Nov 01 09:59:56.196701 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:56.196645    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:58.359385 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:58.359351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 09:59:59.551042 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551007    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"]
Nov 01 09:59:59.551442 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551429    4314 cpu_manager.go:414] "containerMap: removing container" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551443    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551452    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d6599748-433a-4fad-81a1-a989d6df7271" containerName="revoke-keys"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551457    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6599748-433a-4fad-81a1-a989d6df7271" containerName="revoke-keys"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551475    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dfb92746-a22a-4898-a45d-de0a9935931c" containerName="payload-pre-processing"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551480    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb92746-a22a-4898-a45d-de0a9935931c" containerName="payload-pre-processing"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551492    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d834dc53-e017-409a-8d6a-64960a1bc6f2" containerName="payload-processing"
Nov 01 09:59:59.551502 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551496    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d834dc53-e017-409a-8d6a-64960a1bc6f2" containerName="payload-processing"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551516    4314 cpu_manager.go:414] "containerMap: removing container" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551520    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551615    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c20553-56d3-442c-971c-4864a829112e" containerName="main"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551626    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6599748-433a-4fad-81a1-a989d6df7271" containerName="revoke-keys"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551634    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb92746-a22a-4898-a45d-de0a9935931c" containerName="payload-pre-processing"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551642    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d834dc53-e017-409a-8d6a-64960a1bc6f2" containerName="payload-processing"
Nov 01 09:59:59.551753 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.551652    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="54afff9d-b553-41c2-8857-8a17fa342d12" containerName="main"
Nov 01 09:59:59.556010 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.555993    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.606181 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.606158    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 09:59:59.628584 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.628523    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.628747 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.628628    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcz8v\" (UniqueName: \"kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.628861 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.628836    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.652018 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.651984    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 09:59:59.652193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.652077    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 09:59:59.652193 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.652141    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:59:59.655048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.655026    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"]
Nov 01 09:59:59.730396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.730361    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.730396 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.730403    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.730662 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.730426    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcz8v\" (UniqueName: \"kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.731048 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.731017    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.732741 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.732721    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.788809 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.788778    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcz8v\" (UniqueName: \"kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v\") pod \"maas-api-revoke-sub-794f8b20-zc25k\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 09:59:59.868590 ip-10-0-134-93 kubenswrapper[4314]: I1101 09:59:59.868470    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 10:00:00.110580 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.110548    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:00.163014 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.162930    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"]
Nov 01 10:00:00.168350 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.168325    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"]
Nov 01 10:00:00.168481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.168428    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.171862 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.171838    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"]
Nov 01 10:00:00.172001 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.171942    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.175236 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.175216    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.263402 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.262772    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"]
Nov 01 10:00:00.267428 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.267386    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"]
Nov 01 10:00:00.271360 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.271309    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"]
Nov 01 10:00:00.272991 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:00:00.272590    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f974be_0eea_4f4e_9752_4b1ff962ce6e.slice/crio-d4f60ec4b1edb4c7087353ac2479c03bdcb46368c64aee7bfd670bb20c2bb3bd WatchSource:0}: Error finding container d4f60ec4b1edb4c7087353ac2479c03bdcb46368c64aee7bfd670bb20c2bb3bd: Status 404 returned error can't find the container with id d4f60ec4b1edb4c7087353ac2479c03bdcb46368c64aee7bfd670bb20c2bb3bd
Nov 01 10:00:00.273907 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.273888    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"]
Nov 01 10:00:00.337321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.337292    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.337419 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.337326    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8h9c\" (UniqueName: \"kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.337419 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.337408    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.337523 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.337497    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqzlh\" (UniqueName: \"kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.337632 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.337617    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892120-8qmhh\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.337669 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.337639    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq99m\" (UniqueName: \"kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m\") pod \"maas-api-key-cleanup-29892120-8qmhh\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.438497 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.438423    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.438497 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.438478    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqzlh\" (UniqueName: \"kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.438738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.438578    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892120-8qmhh\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.438738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.438608    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq99m\" (UniqueName: \"kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m\") pod \"maas-api-key-cleanup-29892120-8qmhh\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.438738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.438647    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.438738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.438673    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8h9c\" (UniqueName: \"kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.439217 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.439196    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.439384 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.439363    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892120-8qmhh\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.439419 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.439368    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.496208 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.496172    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq99m\" (UniqueName: \"kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m\") pod \"maas-api-key-cleanup-29892120-8qmhh\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.496357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.496332    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqzlh\" (UniqueName: \"kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.496357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.496348    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8h9c\" (UniqueName: \"kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.496463 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.496428    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:00.783088 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.783060    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:00.790281 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.790256    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:00.812975 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.812667    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k" event={"ID":"10f974be-0eea-4f4e-9752-4b1ff962ce6e","Type":"ContainerStarted","Data":"b2b57f90f0837877e3a741dfe27735f33ab3d86fbf0830e8c11fe2a62f80d761"}
Nov 01 10:00:00.813091 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.813017    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k" event={"ID":"10f974be-0eea-4f4e-9752-4b1ff962ce6e","Type":"ContainerStarted","Data":"d4f60ec4b1edb4c7087353ac2479c03bdcb46368c64aee7bfd670bb20c2bb3bd"}
Nov 01 10:00:00.866840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.866811    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"]
Nov 01 10:00:00.869591 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:00:00.869516    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a91ea5_08a9_4482_b3ac_b7b9aa4e0198.slice/crio-a892a3f5a24cdd95e24f0cae348378f36809308f6cce58a48bd0422cdb8980f9 WatchSource:0}: Error finding container a892a3f5a24cdd95e24f0cae348378f36809308f6cce58a48bd0422cdb8980f9: Status 404 returned error can't find the container with id a892a3f5a24cdd95e24f0cae348378f36809308f6cce58a48bd0422cdb8980f9
Nov 01 10:00:00.924092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:00.924042    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k" podStartSLOduration=1.924027985 podStartE2EDuration="1.924027985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:00:00.922470979 +0000 UTC m=+5274.116865537" watchObservedRunningTime="2026-11-01 10:00:00.924027985 +0000 UTC m=+5274.118422540"
Nov 01 10:00:01.158672 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.158642    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"]
Nov 01 10:00:01.159623 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:00:01.159596    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3e2614_1966_407c_8361_47ebb4339b89.slice/crio-eed51c0efd61ccf38a7f5914e5b1cf1e06ffdbdd984141b24c09f69658188d2f WatchSource:0}: Error finding container eed51c0efd61ccf38a7f5914e5b1cf1e06ffdbdd984141b24c09f69658188d2f: Status 404 returned error can't find the container with id eed51c0efd61ccf38a7f5914e5b1cf1e06ffdbdd984141b24c09f69658188d2f
Nov 01 10:00:01.165916 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.165886    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"]
Nov 01 10:00:01.199703 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:00:01.199677    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320d8652_0177_4b38_883a_a2d2a02ced64.slice/crio-7730f636c38a13a62d18b0c0f92442348a87391a9f4fe1b1fe6ecb034e4b8557 WatchSource:0}: Error finding container 7730f636c38a13a62d18b0c0f92442348a87391a9f4fe1b1fe6ecb034e4b8557: Status 404 returned error can't find the container with id 7730f636c38a13a62d18b0c0f92442348a87391a9f4fe1b1fe6ecb034e4b8557
Nov 01 10:00:01.202617 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.202592    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:01.822422 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.822379    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerStarted","Data":"438451ee36d827daa7eb7f70c29acfc4864367693726e3dec0f71a386044d6da"}
Nov 01 10:00:01.822914 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.822491    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerStarted","Data":"7730f636c38a13a62d18b0c0f92442348a87391a9f4fe1b1fe6ecb034e4b8557"}
Nov 01 10:00:01.829315 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.829283    4314 generic.go:356] "Generic (PLEG): container finished" podID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerID="9f1ad62b5494820cfabe0615d09d5948dd3854204d6b4433be021b6ffe4019e4" exitCode=60
Nov 01 10:00:01.829490 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.829373    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" event={"ID":"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198","Type":"ContainerDied","Data":"9f1ad62b5494820cfabe0615d09d5948dd3854204d6b4433be021b6ffe4019e4"}
Nov 01 10:00:01.829490 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.829432    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" event={"ID":"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198","Type":"ContainerStarted","Data":"a892a3f5a24cdd95e24f0cae348378f36809308f6cce58a48bd0422cdb8980f9"}
Nov 01 10:00:01.829642 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.829619    4314 scope.go:122] "RemoveContainer" containerID="9f1ad62b5494820cfabe0615d09d5948dd3854204d6b4433be021b6ffe4019e4"
Nov 01 10:00:01.831086 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.831068    4314 generic.go:356] "Generic (PLEG): container finished" podID="10f974be-0eea-4f4e-9752-4b1ff962ce6e" containerID="b2b57f90f0837877e3a741dfe27735f33ab3d86fbf0830e8c11fe2a62f80d761" exitCode=0
Nov 01 10:00:01.831165 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.831146    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k" event={"ID":"10f974be-0eea-4f4e-9752-4b1ff962ce6e","Type":"ContainerDied","Data":"b2b57f90f0837877e3a741dfe27735f33ab3d86fbf0830e8c11fe2a62f80d761"}
Nov 01 10:00:01.832884 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.832860    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" event={"ID":"4d3e2614-1966-407c-8361-47ebb4339b89","Type":"ContainerStarted","Data":"5073986a0f2251a001e4956733e306832f8ab1bcfb901a6d3bb6cb38384bdaf5"}
Nov 01 10:00:01.832939 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.832908    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" event={"ID":"4d3e2614-1966-407c-8361-47ebb4339b89","Type":"ContainerStarted","Data":"eed51c0efd61ccf38a7f5914e5b1cf1e06ffdbdd984141b24c09f69658188d2f"}
Nov 01 10:00:01.932438 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:01.932384    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" podStartSLOduration=1.932372279 podStartE2EDuration="1.932372279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:00:01.932001583 +0000 UTC m=+5275.126396137" watchObservedRunningTime="2026-11-01 10:00:01.932372279 +0000 UTC m=+5275.126766833"
Nov 01 10:00:02.259569 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.259493    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" podStartSLOduration=2.259478715 podStartE2EDuration="2.259478715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:00:02.25780175 +0000 UTC m=+5275.452196306" watchObservedRunningTime="2026-11-01 10:00:02.259478715 +0000 UTC m=+5275.453873270"
Nov 01 10:00:02.841217 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.841181    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" event={"ID":"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198","Type":"ContainerStarted","Data":"d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044"}
Nov 01 10:00:02.842737 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.842712    4314 generic.go:356] "Generic (PLEG): container finished" podID="4d3e2614-1966-407c-8361-47ebb4339b89" containerID="5073986a0f2251a001e4956733e306832f8ab1bcfb901a6d3bb6cb38384bdaf5" exitCode=60
Nov 01 10:00:02.842848 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.842796    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" event={"ID":"4d3e2614-1966-407c-8361-47ebb4339b89","Type":"ContainerDied","Data":"5073986a0f2251a001e4956733e306832f8ab1bcfb901a6d3bb6cb38384bdaf5"}
Nov 01 10:00:02.843030 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.843011    4314 scope.go:122] "RemoveContainer" containerID="5073986a0f2251a001e4956733e306832f8ab1bcfb901a6d3bb6cb38384bdaf5"
Nov 01 10:00:02.844318 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.844285    4314 generic.go:356] "Generic (PLEG): container finished" podID="320d8652-0177-4b38-883a-a2d2a02ced64" containerID="438451ee36d827daa7eb7f70c29acfc4864367693726e3dec0f71a386044d6da" exitCode=60
Nov 01 10:00:02.844423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.844319    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerDied","Data":"438451ee36d827daa7eb7f70c29acfc4864367693726e3dec0f71a386044d6da"}
Nov 01 10:00:02.844588 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.844570    4314 scope.go:122] "RemoveContainer" containerID="438451ee36d827daa7eb7f70c29acfc4864367693726e3dec0f71a386044d6da"
Nov 01 10:00:02.951841 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.951792    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" podStartSLOduration=2.95177646 podStartE2EDuration="2.95177646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:00:02.951238409 +0000 UTC m=+5276.145632965" watchObservedRunningTime="2026-11-01 10:00:02.95177646 +0000 UTC m=+5276.146171016"
Nov 01 10:00:02.982686 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:02.982661    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 10:00:03.065776 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.065746    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca\") pod \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") "
Nov 01 10:00:03.065913 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.065854    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v\" (UniqueName: \"kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v\") pod \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") "
Nov 01 10:00:03.065968 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.065956    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca\") pod \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\" (UID: \"10f974be-0eea-4f4e-9752-4b1ff962ce6e\") "
Nov 01 10:00:03.066349 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.066320    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca" pod "10f974be-0eea-4f4e-9752-4b1ff962ce6e" (UID: "10f974be-0eea-4f4e-9752-4b1ff962ce6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:03.066555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.066512    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:03.068080 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.068058    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca" pod "10f974be-0eea-4f4e-9752-4b1ff962ce6e" (UID: "10f974be-0eea-4f4e-9752-4b1ff962ce6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:00:03.068171 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.068120    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v" pod "10f974be-0eea-4f4e-9752-4b1ff962ce6e" (UID: "10f974be-0eea-4f4e-9752-4b1ff962ce6e"). InnerVolumeSpecName "kube-api-access-tcz8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:03.167946 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.167919    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f974be-0eea-4f4e-9752-4b1ff962ce6e-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:03.168047 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.167953    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcz8v\" (UniqueName: \"kubernetes.io/projected/10f974be-0eea-4f4e-9752-4b1ff962ce6e-kube-api-access-tcz8v\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:03.851975 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.851934    4314 generic.go:356] "Generic (PLEG): container finished" podID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerID="d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044" exitCode=60
Nov 01 10:00:03.852459 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.852015    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" event={"ID":"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198","Type":"ContainerDied","Data":"d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044"}
Nov 01 10:00:03.852459 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.852083    4314 scope.go:122] "RemoveContainer" containerID="9f1ad62b5494820cfabe0615d09d5948dd3854204d6b4433be021b6ffe4019e4"
Nov 01 10:00:03.852459 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.852306    4314 scope.go:122] "RemoveContainer" containerID="d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044"
Nov 01 10:00:03.852640 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:03.852622    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk_odh-ai-gateway-infra(81a91ea5-08a9-4482-b3ac-b7b9aa4e0198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198"
Nov 01 10:00:03.853427 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.853413    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"
Nov 01 10:00:03.853493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.853438    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k" event={"ID":"10f974be-0eea-4f4e-9752-4b1ff962ce6e","Type":"ContainerDied","Data":"d4f60ec4b1edb4c7087353ac2479c03bdcb46368c64aee7bfd670bb20c2bb3bd"}
Nov 01 10:00:03.853493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.853477    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4f60ec4b1edb4c7087353ac2479c03bdcb46368c64aee7bfd670bb20c2bb3bd"
Nov 01 10:00:03.855398 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.855378    4314 generic.go:356] "Generic (PLEG): container finished" podID="4d3e2614-1966-407c-8361-47ebb4339b89" containerID="a1439a7fc52a65370a706e885a735b8a65cc4d6c09fe9175ad797c32a04b0b6e" exitCode=60
Nov 01 10:00:03.855504 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.855430    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" event={"ID":"4d3e2614-1966-407c-8361-47ebb4339b89","Type":"ContainerDied","Data":"a1439a7fc52a65370a706e885a735b8a65cc4d6c09fe9175ad797c32a04b0b6e"}
Nov 01 10:00:03.855743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.855724    4314 scope.go:122] "RemoveContainer" containerID="a1439a7fc52a65370a706e885a735b8a65cc4d6c09fe9175ad797c32a04b0b6e"
Nov 01 10:00:03.855972 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:03.855947    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892120-8qmhh_odh-ai-gateway-infra(4d3e2614-1966-407c-8361-47ebb4339b89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" podUID="4d3e2614-1966-407c-8361-47ebb4339b89"
Nov 01 10:00:03.857572 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.857528    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerStarted","Data":"d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4"}
Nov 01 10:00:03.925259 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:03.925235    4314 scope.go:122] "RemoveContainer" containerID="5073986a0f2251a001e4956733e306832f8ab1bcfb901a6d3bb6cb38384bdaf5"
Nov 01 10:00:04.865988 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:04.865957    4314 scope.go:122] "RemoveContainer" containerID="d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044"
Nov 01 10:00:04.866409 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:04.866243    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk_odh-ai-gateway-infra(81a91ea5-08a9-4482-b3ac-b7b9aa4e0198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198"
Nov 01 10:00:04.867581 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:04.867560    4314 scope.go:122] "RemoveContainer" containerID="a1439a7fc52a65370a706e885a735b8a65cc4d6c09fe9175ad797c32a04b0b6e"
Nov 01 10:00:04.867801 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:04.867782    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892120-8qmhh_odh-ai-gateway-infra(4d3e2614-1966-407c-8361-47ebb4339b89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" podUID="4d3e2614-1966-407c-8361-47ebb4339b89"
Nov 01 10:00:04.869106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:04.869084    4314 generic.go:356] "Generic (PLEG): container finished" podID="320d8652-0177-4b38-883a-a2d2a02ced64" containerID="d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4" exitCode=60
Nov 01 10:00:04.869190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:04.869164    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerDied","Data":"d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4"}
Nov 01 10:00:04.869235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:04.869221    4314 scope.go:122] "RemoveContainer" containerID="438451ee36d827daa7eb7f70c29acfc4864367693726e3dec0f71a386044d6da"
Nov 01 10:00:04.869334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:04.869320    4314 scope.go:122] "RemoveContainer" containerID="d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4"
Nov 01 10:00:04.869592 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:04.869574    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk_odh-ai-gateway-infra(320d8652-0177-4b38-883a-a2d2a02ced64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" podUID="320d8652-0177-4b38-883a-a2d2a02ced64"
Nov 01 10:00:05.411693 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:05.404385    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:05.879371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:05.879347    4314 scope.go:122] "RemoveContainer" containerID="d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4"
Nov 01 10:00:05.879788 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:05.879613    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk_odh-ai-gateway-infra(320d8652-0177-4b38-883a-a2d2a02ced64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" podUID="320d8652-0177-4b38-883a-a2d2a02ced64"
Nov 01 10:00:07.191892 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:07.191862    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:11.208066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:11.208035    4314 scope.go:122] "RemoveContainer" containerID="54918fb145640ad467fcde6431040ac7b44d4e4a8410f50b35b60397d33ac2bc"
Nov 01 10:00:11.321505 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:11.321468    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:00:11.993893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:11.993856    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:14.424872 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:14.424842    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:14.580723 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:14.580692    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:14.922314 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:14.922280    4314 scope.go:122] "RemoveContainer" containerID="d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044"
Nov 01 10:00:15.800774 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:15.800715    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:15.936333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:15.936303    4314 generic.go:356] "Generic (PLEG): container finished" podID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerID="6d3e82310bd80b8a1408836241ae9a6b4b32e5e5e6d8c11fdc93cd47e3ebd34d" exitCode=60
Nov 01 10:00:15.936514 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:15.936369    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" event={"ID":"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198","Type":"ContainerDied","Data":"6d3e82310bd80b8a1408836241ae9a6b4b32e5e5e6d8c11fdc93cd47e3ebd34d"}
Nov 01 10:00:15.936514 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:15.936432    4314 scope.go:122] "RemoveContainer" containerID="d8c3f04d92b9c01c087fc3b3f8a13d2c4754590cef7da825e1bb293f14217044"
Nov 01 10:00:15.936822 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:15.936802    4314 scope.go:122] "RemoveContainer" containerID="6d3e82310bd80b8a1408836241ae9a6b4b32e5e5e6d8c11fdc93cd47e3ebd34d"
Nov 01 10:00:15.937140 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:15.937107    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk_odh-ai-gateway-infra(81a91ea5-08a9-4482-b3ac-b7b9aa4e0198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198"
Nov 01 10:00:15.949296 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:15.949239    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:16.050077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:16.050044    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"]
Nov 01 10:00:16.093294 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:16.093209    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc190bf5-8md9r"]
Nov 01 10:00:16.208360 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:16.208325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:16.789767 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:16.789733    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:16.925412 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:16.925388    4314 scope.go:122] "RemoveContainer" containerID="a1439a7fc52a65370a706e885a735b8a65cc4d6c09fe9175ad797c32a04b0b6e"
Nov 01 10:00:16.928194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:16.928106    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ea3731-f49e-4700-9df0-0366e462b19a" path="/var/lib/kubelet/pods/e5ea3731-f49e-4700-9df0-0366e462b19a/volumes"
Nov 01 10:00:17.054907 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.054821    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"]
Nov 01 10:00:17.191597 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.191569    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:17.206053 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.206024    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca\") pod \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") "
Nov 01 10:00:17.206241 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.206070    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh\" (UniqueName: \"kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh\") pod \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\" (UID: \"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198\") "
Nov 01 10:00:17.206443 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.206418    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca" pod "81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" (UID: "81a91ea5-08a9-4482-b3ac-b7b9aa4e0198"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:17.208406 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.208382    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh" pod "81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" (UID: "81a91ea5-08a9-4482-b3ac-b7b9aa4e0198"). InnerVolumeSpecName "kube-api-access-lqzlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:17.307273 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.307196    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:17.307495 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.307467    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqzlh\" (UniqueName: \"kubernetes.io/projected/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198-kube-api-access-lqzlh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:17.922350 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.922318    4314 scope.go:122] "RemoveContainer" containerID="d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4"
Nov 01 10:00:17.953410 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.953380    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"
Nov 01 10:00:17.953895 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.953412    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk" event={"ID":"81a91ea5-08a9-4482-b3ac-b7b9aa4e0198","Type":"ContainerDied","Data":"a892a3f5a24cdd95e24f0cae348378f36809308f6cce58a48bd0422cdb8980f9"}
Nov 01 10:00:17.953895 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.953478    4314 scope.go:122] "RemoveContainer" containerID="6d3e82310bd80b8a1408836241ae9a6b4b32e5e5e6d8c11fdc93cd47e3ebd34d"
Nov 01 10:00:17.955496 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.955467    4314 generic.go:356] "Generic (PLEG): container finished" podID="4d3e2614-1966-407c-8361-47ebb4339b89" containerID="5e8dace62c405c73bfbe70fc3b17655ac6550389a4b305e9177fd513bc4de66f" exitCode=60
Nov 01 10:00:17.955610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.955557    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" event={"ID":"4d3e2614-1966-407c-8361-47ebb4339b89","Type":"ContainerDied","Data":"5e8dace62c405c73bfbe70fc3b17655ac6550389a4b305e9177fd513bc4de66f"}
Nov 01 10:00:17.956062 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.955966    4314 scope.go:122] "RemoveContainer" containerID="5e8dace62c405c73bfbe70fc3b17655ac6550389a4b305e9177fd513bc4de66f"
Nov 01 10:00:17.956331 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:17.956307    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892120-8qmhh_odh-ai-gateway-infra(4d3e2614-1966-407c-8361-47ebb4339b89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" podUID="4d3e2614-1966-407c-8361-47ebb4339b89"
Nov 01 10:00:17.963041 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:17.963024    4314 scope.go:122] "RemoveContainer" containerID="a1439a7fc52a65370a706e885a735b8a65cc4d6c09fe9175ad797c32a04b0b6e"
Nov 01 10:00:18.214398 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.214363    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"]
Nov 01 10:00:18.221859 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.221829    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892120-l8kfk"]
Nov 01 10:00:18.927858 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.927774    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" path="/var/lib/kubelet/pods/81a91ea5-08a9-4482-b3ac-b7b9aa4e0198/volumes"
Nov 01 10:00:18.962520 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.962491    4314 generic.go:356] "Generic (PLEG): container finished" podID="320d8652-0177-4b38-883a-a2d2a02ced64" containerID="29cbf723b462d9201c06ef3e455c4ea4b1f009fec6b077c3077ebebdcd5b3ea0" exitCode=60
Nov 01 10:00:18.962960 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.962567    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerDied","Data":"29cbf723b462d9201c06ef3e455c4ea4b1f009fec6b077c3077ebebdcd5b3ea0"}
Nov 01 10:00:18.962960 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.962631    4314 scope.go:122] "RemoveContainer" containerID="d0268ed9303cbfd8386890d0bfb0211fd2b0a6c5e1edd1e61c587a482fe8c5f4"
Nov 01 10:00:18.963082 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:18.962977    4314 scope.go:122] "RemoveContainer" containerID="29cbf723b462d9201c06ef3e455c4ea4b1f009fec6b077c3077ebebdcd5b3ea0"
Nov 01 10:00:18.963279 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:18.963254    4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk_odh-ai-gateway-infra(320d8652-0177-4b38-883a-a2d2a02ced64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" podUID="320d8652-0177-4b38-883a-a2d2a02ced64"
Nov 01 10:00:19.120238 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.120201    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"]
Nov 01 10:00:19.263554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.263517    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:19.328803 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.328766    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca\") pod \"4d3e2614-1966-407c-8361-47ebb4339b89\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") "
Nov 01 10:00:19.328803 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.328808    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m\" (UniqueName: \"kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m\") pod \"4d3e2614-1966-407c-8361-47ebb4339b89\" (UID: \"4d3e2614-1966-407c-8361-47ebb4339b89\") "
Nov 01 10:00:19.329175 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.329144    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca" pod "4d3e2614-1966-407c-8361-47ebb4339b89" (UID: "4d3e2614-1966-407c-8361-47ebb4339b89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:19.330992 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.330973    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m" pod "4d3e2614-1966-407c-8361-47ebb4339b89" (UID: "4d3e2614-1966-407c-8361-47ebb4339b89"). InnerVolumeSpecName "kube-api-access-mq99m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:19.429451 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.429419    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e2614-1966-407c-8361-47ebb4339b89-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:19.429451 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.429446    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq99m\" (UniqueName: \"kubernetes.io/projected/4d3e2614-1966-407c-8361-47ebb4339b89-kube-api-access-mq99m\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:19.972580 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.972524    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh" event={"ID":"4d3e2614-1966-407c-8361-47ebb4339b89","Type":"ContainerDied","Data":"eed51c0efd61ccf38a7f5914e5b1cf1e06ffdbdd984141b24c09f69658188d2f"}
Nov 01 10:00:19.973092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.972609    4314 scope.go:122] "RemoveContainer" containerID="5e8dace62c405c73bfbe70fc3b17655ac6550389a4b305e9177fd513bc4de66f"
Nov 01 10:00:19.973092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:19.972555    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"
Nov 01 10:00:20.080103 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.080068    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"]
Nov 01 10:00:20.126492 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.126456    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"]
Nov 01 10:00:20.132242 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.132209    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892120-8qmhh"]
Nov 01 10:00:20.222555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.222517    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:20.237675 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.237554    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca\") pod \"320d8652-0177-4b38-883a-a2d2a02ced64\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") "
Nov 01 10:00:20.237873 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.237734    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c\" (UniqueName: \"kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c\") pod \"320d8652-0177-4b38-883a-a2d2a02ced64\" (UID: \"320d8652-0177-4b38-883a-a2d2a02ced64\") "
Nov 01 10:00:20.238368 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.238338    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca" pod "320d8652-0177-4b38-883a-a2d2a02ced64" (UID: "320d8652-0177-4b38-883a-a2d2a02ced64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:20.240165 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.240141    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c" pod "320d8652-0177-4b38-883a-a2d2a02ced64" (UID: "320d8652-0177-4b38-883a-a2d2a02ced64"). InnerVolumeSpecName "kube-api-access-n8h9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:20.338893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.338863    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320d8652-0177-4b38-883a-a2d2a02ced64-maas-api-service-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:20.338893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.338889    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8h9c\" (UniqueName: \"kubernetes.io/projected/320d8652-0177-4b38-883a-a2d2a02ced64-kube-api-access-n8h9c\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:20.371753 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.371723    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:20.843092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.843055    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:20.928027 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.927995    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" path="/var/lib/kubelet/pods/4d3e2614-1966-407c-8361-47ebb4339b89/volumes"
Nov 01 10:00:20.982619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.982592    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"
Nov 01 10:00:20.983092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.982585    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk" event={"ID":"320d8652-0177-4b38-883a-a2d2a02ced64","Type":"ContainerDied","Data":"7730f636c38a13a62d18b0c0f92442348a87391a9f4fe1b1fe6ecb034e4b8557"}
Nov 01 10:00:20.983092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:20.982726    4314 scope.go:122] "RemoveContainer" containerID="29cbf723b462d9201c06ef3e455c4ea4b1f009fec6b077c3077ebebdcd5b3ea0"
Nov 01 10:00:21.137090 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:21.137005    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"]
Nov 01 10:00:21.141069 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:21.141041    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892120-qsbfk"]
Nov 01 10:00:21.915386 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:21.915353    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:22.064702 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:22.064658    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:22.927526 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:22.927491    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" path="/var/lib/kubelet/pods/320d8652-0177-4b38-883a-a2d2a02ced64/volumes"
Nov 01 10:00:26.874178 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:26.874148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:30.314313 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:30.314271    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:31.790664 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:31.790622    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:33.045133 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:33.045099    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"]
Nov 01 10:00:33.090767 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:33.090730    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c3e88d-l8bcw"]
Nov 01 10:00:34.927404 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:34.927364    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085c4146-9cce-4ab1-8797-e58847d8fb1f" path="/var/lib/kubelet/pods/085c4146-9cce-4ab1-8797-e58847d8fb1f/volumes"
Nov 01 10:00:36.580177 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:36.580142    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:41.321212 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:00:41.321172    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:00:41.474777 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:41.474742    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:46.197899 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:46.197860    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"]
Nov 01 10:00:46.322167 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:46.322128    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:46.480354 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:46.480325    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:51.120781 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:51.120742    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:52.860298 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:52.860247    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"]
Nov 01 10:00:52.875225 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:52.875188    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:53.325447 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:53.325411    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:55.885823 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:55.885788    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:00:58.376819 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:00:58.376789    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:00.772189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:00.772154    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:01.217225 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.217120    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" podUID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" containerName="main" containerID="cri-o://415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0" gracePeriod=45
Nov 01 10:01:01.282241 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.282206    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" podUID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" containerName="main" probeResult="failure" output="Get \"https://10.132.1.92:8000/ready\": dial tcp 10.132.1.92:8000: connect: connection refused"
Nov 01 10:01:01.466253 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.466229    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 10:01:01.545893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.545866    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs\") pod \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") "
Nov 01 10:01:01.546049 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.545906    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home\") pod \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") "
Nov 01 10:01:01.546049 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.545940    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm\") pod \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") "
Nov 01 10:01:01.546154 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546044    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir\") pod \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") "
Nov 01 10:01:01.546154 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546081    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache\") pod \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") "
Nov 01 10:01:01.546154 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546138    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc\" (UniqueName: \"kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc\") pod \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\" (UID: \"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3\") "
Nov 01 10:01:01.546323 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546234    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home" pod "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" (UID: "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:01.546323 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546291    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir" pod "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" (UID: "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:01.546556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546518    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:01.546556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546524    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache" pod "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" (UID: "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:01.546740 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.546567    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:01.548268 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.548240    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc" pod "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" (UID: "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3"). InnerVolumeSpecName "kube-api-access-2vdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:01.548562 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.548523    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm" pod "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" (UID: "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:01.548693 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.548582    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs" pod "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" (UID: "e70b58ac-0d89-4d44-a0e7-2c4b787b72b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:01:01.647474 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.647434    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:01.647474 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.647464    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:01.647474 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.647473    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:01.647743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:01.647484    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdlc\" (UniqueName: \"kubernetes.io/projected/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3-kube-api-access-2vdlc\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:02.231155 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.231119    4314 generic.go:356] "Generic (PLEG): container finished" podID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" containerID="415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0" exitCode=0
Nov 01 10:01:02.231577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.231173    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"
Nov 01 10:01:02.231577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.231204    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" event={"ID":"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3","Type":"ContainerDied","Data":"415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0"}
Nov 01 10:01:02.231577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.231249    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr" event={"ID":"e70b58ac-0d89-4d44-a0e7-2c4b787b72b3","Type":"ContainerDied","Data":"81242ddfafa34e48769f01428939094d26c9c08dfc3482f39d616a8999b4277a"}
Nov 01 10:01:02.231577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.231267    4314 scope.go:122] "RemoveContainer" containerID="415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0"
Nov 01 10:01:02.241119 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.241102    4314 scope.go:122] "RemoveContainer" containerID="415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0"
Nov 01 10:01:02.241375 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:02.241355    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0\": container with ID starting with 415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0 not found: ID does not exist" containerID="415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0"
Nov 01 10:01:02.241436 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.241383    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0"} err="failed to get container status \"415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0\": rpc error: code = NotFound desc = could not find container \"415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0\": container with ID starting with 415a0d97b4cca190a6e1ebe7d365037720e9645012375f24761b5855cb92e4f0 not found: ID does not exist"
Nov 01 10:01:02.386202 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.386166    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"]
Nov 01 10:01:02.432244 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.432208    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/sub-test-model-a87a1d-kserve-59bc78dd45-4xvlr"]
Nov 01 10:01:02.927946 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:02.927912    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" path="/var/lib/kubelet/pods/e70b58ac-0d89-4d44-a0e7-2c4b787b72b3/volumes"
Nov 01 10:01:05.009623 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:05.009585    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"]
Nov 01 10:01:05.119795 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:05.119761    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:05.124723 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:05.122382    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"]
Nov 01 10:01:05.389147 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:05.389074    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:05.538284 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:05.538247    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:05.752095 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:05.752066    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:07.879651 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:07.879609    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb" podUID="fee9992c-2ff3-4033-86b6-e6303cc588e0" containerName="main" containerID="cri-o://6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5" gracePeriod=45
Nov 01 10:01:08.122746 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.122722    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 10:01:08.206863 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.206780    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir\") pod \"fee9992c-2ff3-4033-86b6-e6303cc588e0\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") "
Nov 01 10:01:08.207038 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.206883    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm\") pod \"fee9992c-2ff3-4033-86b6-e6303cc588e0\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") "
Nov 01 10:01:08.207038 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207009    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home\") pod \"fee9992c-2ff3-4033-86b6-e6303cc588e0\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") "
Nov 01 10:01:08.207177 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207032    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir" pod "fee9992c-2ff3-4033-86b6-e6303cc588e0" (UID: "fee9992c-2ff3-4033-86b6-e6303cc588e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:08.207177 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207085    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs\") pod \"fee9992c-2ff3-4033-86b6-e6303cc588e0\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") "
Nov 01 10:01:08.207177 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207117    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9\" (UniqueName: \"kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9\") pod \"fee9992c-2ff3-4033-86b6-e6303cc588e0\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") "
Nov 01 10:01:08.207177 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207150    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache\") pod \"fee9992c-2ff3-4033-86b6-e6303cc588e0\" (UID: \"fee9992c-2ff3-4033-86b6-e6303cc588e0\") "
Nov 01 10:01:08.207372 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207261    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home" pod "fee9992c-2ff3-4033-86b6-e6303cc588e0" (UID: "fee9992c-2ff3-4033-86b6-e6303cc588e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:08.207501 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207471    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache" pod "fee9992c-2ff3-4033-86b6-e6303cc588e0" (UID: "fee9992c-2ff3-4033-86b6-e6303cc588e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:08.207633 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207614    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:08.207684 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207638    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:08.207684 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.207654    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:08.209016 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.208993    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm" pod "fee9992c-2ff3-4033-86b6-e6303cc588e0" (UID: "fee9992c-2ff3-4033-86b6-e6303cc588e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:08.209212 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.209194    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9" pod "fee9992c-2ff3-4033-86b6-e6303cc588e0" (UID: "fee9992c-2ff3-4033-86b6-e6303cc588e0"). InnerVolumeSpecName "kube-api-access-sxtz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:08.209271 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.209231    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs" pod "fee9992c-2ff3-4033-86b6-e6303cc588e0" (UID: "fee9992c-2ff3-4033-86b6-e6303cc588e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:01:08.267846 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.267807    4314 generic.go:356] "Generic (PLEG): container finished" podID="fee9992c-2ff3-4033-86b6-e6303cc588e0" containerID="6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5" exitCode=0
Nov 01 10:01:08.268022 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.267858    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"
Nov 01 10:01:08.268022 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.267870    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb" event={"ID":"fee9992c-2ff3-4033-86b6-e6303cc588e0","Type":"ContainerDied","Data":"6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5"}
Nov 01 10:01:08.268022 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.267909    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb" event={"ID":"fee9992c-2ff3-4033-86b6-e6303cc588e0","Type":"ContainerDied","Data":"d3d3230745ae149912dde07237ac95aeccc4c80a27b885df63b03b422c2e9ccd"}
Nov 01 10:01:08.268022 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.267930    4314 scope.go:122] "RemoveContainer" containerID="6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5"
Nov 01 10:01:08.278215 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.278194    4314 scope.go:122] "RemoveContainer" containerID="6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5"
Nov 01 10:01:08.278514 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:08.278489    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5\": container with ID starting with 6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5 not found: ID does not exist" containerID="6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5"
Nov 01 10:01:08.278627 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.278522    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5"} err="failed to get container status \"6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5\": rpc error: code = NotFound desc = could not find container \"6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5\": container with ID starting with 6d04311d6f64cff3e5d89247d5dfdf05ca4c7aa9672a232d773a22d9f8e649f5 not found: ID does not exist"
Nov 01 10:01:08.308840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.308816    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee9992c-2ff3-4033-86b6-e6303cc588e0-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:08.308840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.308839    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee9992c-2ff3-4033-86b6-e6303cc588e0-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:08.308981 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.308849    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtz9\" (UniqueName: \"kubernetes.io/projected/fee9992c-2ff3-4033-86b6-e6303cc588e0-kube-api-access-sxtz9\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:08.422353 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.422319    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"]
Nov 01 10:01:08.426995 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.426970    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-65aeef/sub-test-model-65aeef-kserve-5bcbc599c-kpzxb"]
Nov 01 10:01:08.927282 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:08.927240    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee9992c-2ff3-4033-86b6-e6303cc588e0" path="/var/lib/kubelet/pods/fee9992c-2ff3-4033-86b6-e6303cc588e0/volumes"
Nov 01 10:01:10.010004 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:10.009961    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerName="payload-processing" containerID="cri-o://15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938" gracePeriod=25
Nov 01 10:01:10.123409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:10.123361    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerName="payload-pre-processing" containerID="cri-o://dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96" gracePeriod=25
Nov 01 10:01:10.709115 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:10.709084    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:10.947022 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:10.946986    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:11.260110 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.260082    4314 scope.go:122] "RemoveContainer" containerID="a8ba1074635eef0b84f36ef9597fdaf07d9050d4276f0d2a2f8cd2ab7bd6cedb"
Nov 01 10:01:11.269109 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.269087    4314 scope.go:122] "RemoveContainer" containerID="fd959b94402b9c561fab96b389f0f045fb0cec141785d78d84ac79f455c6c42d"
Nov 01 10:01:11.277773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.277753    4314 scope.go:122] "RemoveContainer" containerID="7952f6e0911999a624cd4aaffd8271aedf8925fc0bffe552fe419e3ddb0b338a"
Nov 01 10:01:11.324424 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:11.324384    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:01:11.445200 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.445166    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:11.591728 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.591599    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:01:11.591728 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.591647    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:01:11.596815 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:11.596788    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:15.495772 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:15.495736    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:19.777499 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:19.777415    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:21.514514 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:21.514479    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:21.592298 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:21.592254    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:01:21.592484 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:21.592370    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:01:26.019048 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:26.019013    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:26.173035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:26.172997    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:26.371752 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:26.371674    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:26.640206 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:26.640116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:26.790348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:26.790312    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:28.254484 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:28.254448    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:28.730945 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:28.730922    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"]
Nov 01 10:01:28.731129 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:28.731109    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" containerID="cri-o://f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7" gracePeriod=30
Nov 01 10:01:28.744149 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:28.744123    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:29.557669 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:29.557635    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:30.266025 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.265965    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 10:01:30.363904 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.363878    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 10:01:30.390617 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.390581    4314 generic.go:356] "Generic (PLEG): container finished" podID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerID="dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96" exitCode=0
Nov 01 10:01:30.390778 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.390633    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"
Nov 01 10:01:30.390778 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.390664    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" event={"ID":"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae","Type":"ContainerDied","Data":"dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96"}
Nov 01 10:01:30.390778 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.390712    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs" event={"ID":"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae","Type":"ContainerDied","Data":"2f405541b28cb3a5e7f0f1829d827d6eda3b021400313032293801f8f603b1fe"}
Nov 01 10:01:30.390778 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.390740    4314 scope.go:122] "RemoveContainer" containerID="dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96"
Nov 01 10:01:30.392052 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.392029    4314 generic.go:356] "Generic (PLEG): container finished" podID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerID="15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938" exitCode=0
Nov 01 10:01:30.392182 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.392071    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"
Nov 01 10:01:30.392182 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.392118    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" event={"ID":"e362b181-5253-4db8-b41a-eb53e2bde6ed","Type":"ContainerDied","Data":"15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938"}
Nov 01 10:01:30.392182 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.392163    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5" event={"ID":"e362b181-5253-4db8-b41a-eb53e2bde6ed","Type":"ContainerDied","Data":"c1bcee3d50cac44e2b71b15591caba88c7abe574f05ee3d0a49de8267e4b1ac4"}
Nov 01 10:01:30.399398 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.399380    4314 scope.go:122] "RemoveContainer" containerID="dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96"
Nov 01 10:01:30.399664 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:30.399632    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96\": container with ID starting with dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96 not found: ID does not exist" containerID="dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96"
Nov 01 10:01:30.399766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.399662    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96"} err="failed to get container status \"dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96\": rpc error: code = NotFound desc = could not find container \"dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96\": container with ID starting with dcff14124ea830c5348f43ec29007baf36299b67fbfe3e9e58631f1f89bdbf96 not found: ID does not exist"
Nov 01 10:01:30.399766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.399680    4314 scope.go:122] "RemoveContainer" containerID="15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938"
Nov 01 10:01:30.409429 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.409412    4314 scope.go:122] "RemoveContainer" containerID="15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938"
Nov 01 10:01:30.409706 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:30.409687    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938\": container with ID starting with 15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938 not found: ID does not exist" containerID="15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938"
Nov 01 10:01:30.409772 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.409712    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938"} err="failed to get container status \"15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938\": rpc error: code = NotFound desc = could not find container \"15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938\": container with ID starting with 15e66b94663ed955760bcb13f2379da7e951483c6e8ebc413f3ed122df877938 not found: ID does not exist"
Nov 01 10:01:30.426031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.426010    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume\") pod \"e362b181-5253-4db8-b41a-eb53e2bde6ed\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") "
Nov 01 10:01:30.426147 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.426132    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l\" (UniqueName: \"kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l\") pod \"e362b181-5253-4db8-b41a-eb53e2bde6ed\" (UID: \"e362b181-5253-4db8-b41a-eb53e2bde6ed\") "
Nov 01 10:01:30.426350 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.426330    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume" pod "e362b181-5253-4db8-b41a-eb53e2bde6ed" (UID: "e362b181-5253-4db8-b41a-eb53e2bde6ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:01:30.426465 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.426447    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362b181-5253-4db8-b41a-eb53e2bde6ed-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:30.428189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.428171    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l" pod "e362b181-5253-4db8-b41a-eb53e2bde6ed" (UID: "e362b181-5253-4db8-b41a-eb53e2bde6ed"). InnerVolumeSpecName "kube-api-access-rgb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:30.527344 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.527268    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv\" (UniqueName: \"kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv\") pod \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") "
Nov 01 10:01:30.527344 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.527315    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume\") pod \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\" (UID: \"5c6f5239-c5fa-4a07-b0ae-626c2be2cfae\") "
Nov 01 10:01:30.527585 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.527470    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgb6l\" (UniqueName: \"kubernetes.io/projected/e362b181-5253-4db8-b41a-eb53e2bde6ed-kube-api-access-rgb6l\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:30.527750 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.527728    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume" pod "5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" (UID: "5c6f5239-c5fa-4a07-b0ae-626c2be2cfae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:01:30.529394 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.529375    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv" pod "5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" (UID: "5c6f5239-c5fa-4a07-b0ae-626c2be2cfae"). InnerVolumeSpecName "kube-api-access-d6lfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:30.628123 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.628094    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lfv\" (UniqueName: \"kubernetes.io/projected/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-kube-api-access-d6lfv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:30.628123 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.628118    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:30.657876 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.657847    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:30.806731 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.806644    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"]
Nov 01 10:01:30.808932 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.808904    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a87a1d-5b78bc58c5-gqvh5"]
Nov 01 10:01:30.911098 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.911064    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"]
Nov 01 10:01:30.915677 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.915652    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a87a1d-5d86df84f9-vdrhs"]
Nov 01 10:01:30.927737 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.927708    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" path="/var/lib/kubelet/pods/5c6f5239-c5fa-4a07-b0ae-626c2be2cfae/volumes"
Nov 01 10:01:30.928102 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:30.928090    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" path="/var/lib/kubelet/pods/e362b181-5253-4db8-b41a-eb53e2bde6ed/volumes"
Nov 01 10:01:31.489775 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:31.489740    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:33.011494 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:33.011454    4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 10:01:33.011917 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:33.011512    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 10:01:33.981163 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:33.981137    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 10:01:34.163938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.163857    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.163938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.163903    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.163939    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.163980    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164015    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164049    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164127    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164133    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164234    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164273    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo\") pod \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\" (UID: \"c07cb848-0d0f-4f34-8b25-49a143dee3d1\") "
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164317    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.164467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164333    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.165031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164563    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.165031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164632    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.165031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164654    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.165031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164671    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.165031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.164683    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:01:34.166309 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.166278    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.166449 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.166401    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:34.166634 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.166614    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "kube-api-access-nm49z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:34.166699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.166620    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo" pod "c07cb848-0d0f-4f34-8b25-49a143dee3d1" (UID: "c07cb848-0d0f-4f34-8b25-49a143dee3d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:01:34.266194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.266155    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.266194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.266188    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.266194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.266200    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm49z\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-kube-api-access-nm49z\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.266452 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.266211    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.266452 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.266220    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.266452 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.266228    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07cb848-0d0f-4f34-8b25-49a143dee3d1-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.416466 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.416384    4314 generic.go:356] "Generic (PLEG): container finished" podID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerID="f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7" exitCode=0
Nov 01 10:01:34.416466 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.416445    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"
Nov 01 10:01:34.416743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.416479    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" event={"ID":"c07cb848-0d0f-4f34-8b25-49a143dee3d1","Type":"ContainerDied","Data":"f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7"}
Nov 01 10:01:34.416743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.416565    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq" event={"ID":"c07cb848-0d0f-4f34-8b25-49a143dee3d1","Type":"ContainerDied","Data":"13e4254142824e54d3994f6e7b9ca19450e79c8170c0d0fc689beeea0a355401"}
Nov 01 10:01:34.416743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.416598    4314 scope.go:122] "RemoveContainer" containerID="f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7"
Nov 01 10:01:34.426552 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.426513    4314 scope.go:122] "RemoveContainer" containerID="c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"
Nov 01 10:01:34.434998 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.434979    4314 scope.go:122] "RemoveContainer" containerID="f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7"
Nov 01 10:01:34.435244 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:34.435227    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7\": container with ID starting with f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7 not found: ID does not exist" containerID="f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7"
Nov 01 10:01:34.435295 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.435253    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7"} err="failed to get container status \"f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7\": rpc error: code = NotFound desc = could not find container \"f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7\": container with ID starting with f92bbaac7806541b3b31097fcdfb95a05984eec618d940dbaaf2b62805758ad7 not found: ID does not exist"
Nov 01 10:01:34.435295 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.435270    4314 scope.go:122] "RemoveContainer" containerID="c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"
Nov 01 10:01:34.435480 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:34.435467    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb\": container with ID starting with c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb not found: ID does not exist" containerID="c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"
Nov 01 10:01:34.435527 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.435483    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb"} err="failed to get container status \"c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb\": rpc error: code = NotFound desc = could not find container \"c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb\": container with ID starting with c77f0a989b2f085f33a0b022316ea6ed67ba45ee64aeecb09fae9637846808cb not found: ID does not exist"
Nov 01 10:01:34.570178 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.570144    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"]
Nov 01 10:01:34.616079 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.616052    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-zl2wq"]
Nov 01 10:01:34.927415 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:34.927381    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" path="/var/lib/kubelet/pods/c07cb848-0d0f-4f34-8b25-49a143dee3d1/volumes"
Nov 01 10:01:35.121479 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:35.121392    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"]
Nov 01 10:01:35.374975 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:35.374902    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:35.523377 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:35.523340    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:35.996334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:35.996293    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:36.626237 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:36.626206    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:37.978365 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:37.978334    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:40.122889 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:40.122852    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" containerName="payload-pre-processing" containerID="cri-o://569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b" gracePeriod=25
Nov 01 10:01:41.324286 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:01:41.324247    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:01:42.690093 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:42.690060    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:42.860713 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:42.860678    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:43.018050 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:43.018017    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:44.190750 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:44.190717    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:44.678092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:44.678061    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:45.142902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.142861    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:45.802023 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.801987    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"]
Nov 01 10:01:45.802416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802377    4314 cpu_manager.go:414] "containerMap: removing container" podUID="10f974be-0eea-4f4e-9752-4b1ff962ce6e" containerName="revoke-keys"
Nov 01 10:01:45.802416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802388    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f974be-0eea-4f4e-9752-4b1ff962ce6e" containerName="revoke-keys"
Nov 01 10:01:45.802416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802397    4314 cpu_manager.go:414] "containerMap: removing container" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:45.802416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802404    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:45.802416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802413    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerName="payload-processing"
Nov 01 10:01:45.802416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802418    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerName="payload-processing"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802427    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerName="payload-pre-processing"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802433    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerName="payload-pre-processing"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802442    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802447    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802453    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802457    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802463    4314 cpu_manager.go:414] "containerMap: removing container" podUID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" containerName="main"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802467    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" containerName="main"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802474    4314 cpu_manager.go:414] "containerMap: removing container" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802478    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802490    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802494    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802506    4314 cpu_manager.go:414] "containerMap: removing container" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802510    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802520    4314 cpu_manager.go:414] "containerMap: removing container" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802524    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802547    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fee9992c-2ff3-4033-86b6-e6303cc588e0" containerName="main"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802552    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee9992c-2ff3-4033-86b6-e6303cc588e0" containerName="main"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802563    4314 cpu_manager.go:414] "containerMap: removing container" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802567    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802618    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802625    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802632    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802636    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee9992c-2ff3-4033-86b6-e6303cc588e0" containerName="main"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802644    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802650    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:45.802646 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802656    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f974be-0eea-4f4e-9752-4b1ff962ce6e" containerName="revoke-keys"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802662    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e362b181-5253-4db8-b41a-eb53e2bde6ed" containerName="payload-processing"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802667    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6f5239-c5fa-4a07-b0ae-626c2be2cfae" containerName="payload-pre-processing"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802672    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802677    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802682    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70b58ac-0d89-4d44-a0e7-2c4b787b72b3" containerName="main"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802687    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="320d8652-0177-4b38-883a-a2d2a02ced64" containerName="cleanup"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802694    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy"
Nov 01 10:01:45.803493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.802700    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy"
Nov 01 10:01:45.807387 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.807368    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.870680 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.870652    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7lg\" (UniqueName: \"kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.870821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.870687    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.870821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.870713    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.902696 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.902662    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:01:45.902860 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.902718    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:01:45.902932 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.902855    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:01:45.902978 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.902951    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 10:01:45.905340 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.905324    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"]
Nov 01 10:01:45.972146 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.972114    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7lg\" (UniqueName: \"kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.972146 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.972148    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.972400 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.972177    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.972919 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.972896    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:45.974602 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:45.974582    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:46.029520 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:46.029497    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7lg\" (UniqueName: \"kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg\") pod \"maas-api-revoke-sub-dbebde98-tbbxs\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:46.120000 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:46.119930    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:46.490028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:46.490002    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"]
Nov 01 10:01:46.491719 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:01:46.491691    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc409c066_08cf_4e4e_aeef_ad4359c5da5e.slice/crio-f1aedd50bd18b61efa64d59d56b0c068b97b4643fd4e69643058a8ce9a42f554 WatchSource:0}: Error finding container f1aedd50bd18b61efa64d59d56b0c068b97b4643fd4e69643058a8ce9a42f554: Status 404 returned error can't find the container with id f1aedd50bd18b61efa64d59d56b0c068b97b4643fd4e69643058a8ce9a42f554
Nov 01 10:01:47.492198 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:47.492167    4314 generic.go:356] "Generic (PLEG): container finished" podID="c409c066-08cf-4e4e-aeef-ad4359c5da5e" containerID="584ab4de6cab4a662e691a800769e5264b086f1b87823d9e100af60ade8d371c" exitCode=0
Nov 01 10:01:47.492649 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:47.492253    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs" event={"ID":"c409c066-08cf-4e4e-aeef-ad4359c5da5e","Type":"ContainerDied","Data":"584ab4de6cab4a662e691a800769e5264b086f1b87823d9e100af60ade8d371c"}
Nov 01 10:01:47.492649 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:47.492330    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs" event={"ID":"c409c066-08cf-4e4e-aeef-ad4359c5da5e","Type":"ContainerStarted","Data":"f1aedd50bd18b61efa64d59d56b0c068b97b4643fd4e69643058a8ce9a42f554"}
Nov 01 10:01:48.162111 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.162078    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:48.581197 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.581166    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:01:48.622161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.622135    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:48.697491 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.697452    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca\") pod \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") "
Nov 01 10:01:48.697702 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.697512    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca\") pod \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") "
Nov 01 10:01:48.697702 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.697628    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg\" (UniqueName: \"kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg\") pod \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\" (UID: \"c409c066-08cf-4e4e-aeef-ad4359c5da5e\") "
Nov 01 10:01:48.697903 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.697880    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca" pod "c409c066-08cf-4e4e-aeef-ad4359c5da5e" (UID: "c409c066-08cf-4e4e-aeef-ad4359c5da5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:01:48.699584 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.699558    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca" pod "c409c066-08cf-4e4e-aeef-ad4359c5da5e" (UID: "c409c066-08cf-4e4e-aeef-ad4359c5da5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:01:48.699690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.699659    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg" pod "c409c066-08cf-4e4e-aeef-ad4359c5da5e" (UID: "c409c066-08cf-4e4e-aeef-ad4359c5da5e"). InnerVolumeSpecName "kube-api-access-bb7lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:48.799008 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.798905    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7lg\" (UniqueName: \"kubernetes.io/projected/c409c066-08cf-4e4e-aeef-ad4359c5da5e-kube-api-access-bb7lg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:48.799008 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.798953    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:48.799008 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:48.798967    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c409c066-08cf-4e4e-aeef-ad4359c5da5e-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:49.502938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:49.502903    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"
Nov 01 10:01:49.502938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:49.502932    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs" event={"ID":"c409c066-08cf-4e4e-aeef-ad4359c5da5e","Type":"ContainerDied","Data":"f1aedd50bd18b61efa64d59d56b0c068b97b4643fd4e69643058a8ce9a42f554"}
Nov 01 10:01:49.503162 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:49.502980    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1aedd50bd18b61efa64d59d56b0c068b97b4643fd4e69643058a8ce9a42f554"
Nov 01 10:01:50.234843 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:50.234808    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:52.271580 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:52.271527    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:53.107654 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:53.107621    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:58.006297 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.006255    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:58.339375 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339287    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"]
Nov 01 10:01:58.339712 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339695    4314 cpu_manager.go:414] "containerMap: removing container" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339714    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339727    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339732    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e2614-1966-407c-8361-47ebb4339b89" containerName="cleanup"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339739    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c409c066-08cf-4e4e-aeef-ad4359c5da5e" containerName="revoke-keys"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339744    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c409c066-08cf-4e4e-aeef-ad4359c5da5e" containerName="revoke-keys"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339757    4314 cpu_manager.go:414] "containerMap: removing container" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy"
Nov 01 10:01:58.339770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339761    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07cb848-0d0f-4f34-8b25-49a143dee3d1" containerName="istio-proxy"
Nov 01 10:01:58.340006 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339834    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a91ea5-08a9-4482-b3ac-b7b9aa4e0198" containerName="cleanup"
Nov 01 10:01:58.340006 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.339846    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c409c066-08cf-4e4e-aeef-ad4359c5da5e" containerName="revoke-keys"
Nov 01 10:01:58.342985 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.342967    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.388954 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.388919    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.389140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.388960    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.389140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.389082    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kvj\" (UniqueName: \"kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.394077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.393879    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:01:58.394077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.393930    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:01:58.394077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.393921    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:01:58.440053 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.440022    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 10:01:58.442564 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.442517    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"]
Nov 01 10:01:58.490016 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.489979    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kvj\" (UniqueName: \"kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.490218 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.490069    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.490218 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.490106    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.490954 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.490925    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.492612 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.492594    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.547657 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.547629    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kvj\" (UniqueName: \"kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj\") pod \"maas-api-revoke-sub-2e171d6e-p6nhx\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.581045 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.581011    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:01:58.656917 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.656843    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:01:58.835363 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.835324    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"]
Nov 01 10:01:58.853021 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:58.852985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:59.022693 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:59.022671    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"]
Nov 01 10:01:59.025021 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:01:59.024991    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc11a28_54a9_4e95_a393_533cdff0d576.slice/crio-bcf756bfa465c102a982ba689428399becd735460c0d82ea60e7e49445918fdb WatchSource:0}: Error finding container bcf756bfa465c102a982ba689428399becd735460c0d82ea60e7e49445918fdb: Status 404 returned error can't find the container with id bcf756bfa465c102a982ba689428399becd735460c0d82ea60e7e49445918fdb
Nov 01 10:01:59.209296 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:59.209253    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:59.358908 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:59.358817    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:01:59.557005 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:59.556966    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx" event={"ID":"cbc11a28-54a9-4e95-a393-533cdff0d576","Type":"ContainerStarted","Data":"5a2bb06a39264b573612ad74f09caee9e67d106d46016e20d9f54b45af7479a0"}
Nov 01 10:01:59.557190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:59.557027    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx" event={"ID":"cbc11a28-54a9-4e95-a393-533cdff0d576","Type":"ContainerStarted","Data":"bcf756bfa465c102a982ba689428399becd735460c0d82ea60e7e49445918fdb"}
Nov 01 10:01:59.667414 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:01:59.667294    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx" podStartSLOduration=1.667272272 podStartE2EDuration="1.667272272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:01:59.665180323 +0000 UTC m=+5392.859574890" watchObservedRunningTime="2026-11-01 10:01:59.667272272 +0000 UTC m=+5392.861666828"
Nov 01 10:02:00.360019 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.359991    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 10:02:00.409337 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.409273    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume\") pod \"2757fbbc-4757-41f2-b7b9-da1397533569\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") "
Nov 01 10:02:00.409483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.409425    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q\" (UniqueName: \"kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q\") pod \"2757fbbc-4757-41f2-b7b9-da1397533569\" (UID: \"2757fbbc-4757-41f2-b7b9-da1397533569\") "
Nov 01 10:02:00.409765 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.409738    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume" pod "2757fbbc-4757-41f2-b7b9-da1397533569" (UID: "2757fbbc-4757-41f2-b7b9-da1397533569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:02:00.411439 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.411417    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q" pod "2757fbbc-4757-41f2-b7b9-da1397533569" (UID: "2757fbbc-4757-41f2-b7b9-da1397533569"). InnerVolumeSpecName "kube-api-access-4fn2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:00.510464 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.510432    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fn2q\" (UniqueName: \"kubernetes.io/projected/2757fbbc-4757-41f2-b7b9-da1397533569-kube-api-access-4fn2q\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:00.510464 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.510464    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2757fbbc-4757-41f2-b7b9-da1397533569-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:00.563987 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.563953    4314 generic.go:356] "Generic (PLEG): container finished" podID="cbc11a28-54a9-4e95-a393-533cdff0d576" containerID="5a2bb06a39264b573612ad74f09caee9e67d106d46016e20d9f54b45af7479a0" exitCode=0
Nov 01 10:02:00.564162 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.564036    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx" event={"ID":"cbc11a28-54a9-4e95-a393-533cdff0d576","Type":"ContainerDied","Data":"5a2bb06a39264b573612ad74f09caee9e67d106d46016e20d9f54b45af7479a0"}
Nov 01 10:02:00.565479 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.565453    4314 generic.go:356] "Generic (PLEG): container finished" podID="2757fbbc-4757-41f2-b7b9-da1397533569" containerID="569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b" exitCode=0
Nov 01 10:02:00.565619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.565493    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"
Nov 01 10:02:00.565619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.565526    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" event={"ID":"2757fbbc-4757-41f2-b7b9-da1397533569","Type":"ContainerDied","Data":"569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b"}
Nov 01 10:02:00.565619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.565584    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr" event={"ID":"2757fbbc-4757-41f2-b7b9-da1397533569","Type":"ContainerDied","Data":"1a286a759ee2dffcf3301cafe28807400b8ad95a9e8645ef095a3fff89c90fea"}
Nov 01 10:02:00.565619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.565606    4314 scope.go:122] "RemoveContainer" containerID="569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b"
Nov 01 10:02:00.574856 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.574829    4314 scope.go:122] "RemoveContainer" containerID="569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b"
Nov 01 10:02:00.575126 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:02:00.575107    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b\": container with ID starting with 569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b not found: ID does not exist" containerID="569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b"
Nov 01 10:02:00.575176 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.575136    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b"} err="failed to get container status \"569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b\": rpc error: code = NotFound desc = could not find container \"569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b\": container with ID starting with 569b702d7716525b28f2baf738b28b33f2f9d768cff4e6c5d0e19320e2dce57b not found: ID does not exist"
Nov 01 10:02:00.777189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.777155    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"]
Nov 01 10:02:00.781297 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.781269    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-65aeef-798b7dc967-dngvr"]
Nov 01 10:02:00.929216 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:00.929180    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" path="/var/lib/kubelet/pods/2757fbbc-4757-41f2-b7b9-da1397533569/volumes"
Nov 01 10:02:01.702866 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.702843    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:02:01.824311 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.824236    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj\" (UniqueName: \"kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj\") pod \"cbc11a28-54a9-4e95-a393-533cdff0d576\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") "
Nov 01 10:02:01.824481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.824362    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca\") pod \"cbc11a28-54a9-4e95-a393-533cdff0d576\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") "
Nov 01 10:02:01.824481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.824463    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca\") pod \"cbc11a28-54a9-4e95-a393-533cdff0d576\" (UID: \"cbc11a28-54a9-4e95-a393-533cdff0d576\") "
Nov 01 10:02:01.824770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.824744    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca" pod "cbc11a28-54a9-4e95-a393-533cdff0d576" (UID: "cbc11a28-54a9-4e95-a393-533cdff0d576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:02:01.826496 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.826476    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj" pod "cbc11a28-54a9-4e95-a393-533cdff0d576" (UID: "cbc11a28-54a9-4e95-a393-533cdff0d576"). InnerVolumeSpecName "kube-api-access-s2kvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:01.826608 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.826477    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca" pod "cbc11a28-54a9-4e95-a393-533cdff0d576" (UID: "cbc11a28-54a9-4e95-a393-533cdff0d576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:02:01.925673 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.925644    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kvj\" (UniqueName: \"kubernetes.io/projected/cbc11a28-54a9-4e95-a393-533cdff0d576-kube-api-access-s2kvj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:01.925673 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.925671    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:01.925828 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:01.925685    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc11a28-54a9-4e95-a393-533cdff0d576-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:02.578732 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:02.578701    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"
Nov 01 10:02:02.578901 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:02.578737    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx" event={"ID":"cbc11a28-54a9-4e95-a393-533cdff0d576","Type":"ContainerDied","Data":"bcf756bfa465c102a982ba689428399becd735460c0d82ea60e7e49445918fdb"}
Nov 01 10:02:02.578901 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:02.578793    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf756bfa465c102a982ba689428399becd735460c0d82ea60e7e49445918fdb"
Nov 01 10:02:03.102015 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:03.101987    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:03.835713 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:03.835646    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" podUID="dddb67ed-071b-4181-890f-56355834104f" containerName="payload-processing" containerID="cri-o://467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8" gracePeriod=25
Nov 01 10:02:05.206762 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:05.206727    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:08.898405 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:08.898363    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" podUID="dddb67ed-071b-4181-890f-56355834104f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:02:10.899049 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:10.898985    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:11.321862 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:02:11.321815    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:02:11.330246 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:11.330218    4314 scope.go:122] "RemoveContainer" containerID="be2ced4bf9aee5bfc34307ccc76cc732286c9caf72b68f875cca00365d22ab19"
Nov 01 10:02:11.338688 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:11.338659    4314 scope.go:122] "RemoveContainer" containerID="f9ebec0fca3bc5d19d91e281de0c95b8cfda5e825e935bdec8978d375e0560b4"
Nov 01 10:02:12.462140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:12.462100    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:16.752244 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:16.752212    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:16.998163 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:16.998134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:18.898233 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:18.898198    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" podUID="dddb67ed-071b-4181-890f-56355834104f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:02:24.084812 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.084783    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 10:02:24.237506 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.237478    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5\" (UniqueName: \"kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5\") pod \"dddb67ed-071b-4181-890f-56355834104f\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") "
Nov 01 10:02:24.237726 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.237612    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume\") pod \"dddb67ed-071b-4181-890f-56355834104f\" (UID: \"dddb67ed-071b-4181-890f-56355834104f\") "
Nov 01 10:02:24.237957 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.237934    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume" pod "dddb67ed-071b-4181-890f-56355834104f" (UID: "dddb67ed-071b-4181-890f-56355834104f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:02:24.239594 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.239568    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5" pod "dddb67ed-071b-4181-890f-56355834104f" (UID: "dddb67ed-071b-4181-890f-56355834104f"). InnerVolumeSpecName "kube-api-access-9gth5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:24.339301 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.339257    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gth5\" (UniqueName: \"kubernetes.io/projected/dddb67ed-071b-4181-890f-56355834104f-kube-api-access-9gth5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:24.339301 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.339289    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddb67ed-071b-4181-890f-56355834104f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:24.699983 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.699894    4314 generic.go:356] "Generic (PLEG): container finished" podID="dddb67ed-071b-4181-890f-56355834104f" containerID="467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8" exitCode=0
Nov 01 10:02:24.699983 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.699947    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"
Nov 01 10:02:24.700235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.699979    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" event={"ID":"dddb67ed-071b-4181-890f-56355834104f","Type":"ContainerDied","Data":"467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8"}
Nov 01 10:02:24.700235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.700039    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg" event={"ID":"dddb67ed-071b-4181-890f-56355834104f","Type":"ContainerDied","Data":"88c72a992c13e9d7d8eaa495bcb4a9f93cdf87cb5378a4a0479fde35bc7a6589"}
Nov 01 10:02:24.700235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.700064    4314 scope.go:122] "RemoveContainer" containerID="467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8"
Nov 01 10:02:24.709740 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.709721    4314 scope.go:122] "RemoveContainer" containerID="467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8"
Nov 01 10:02:24.710007 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:02:24.709988    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8\": container with ID starting with 467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8 not found: ID does not exist" containerID="467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8"
Nov 01 10:02:24.710099 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.710020    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8"} err="failed to get container status \"467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8\": rpc error: code = NotFound desc = could not find container \"467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8\": container with ID starting with 467718aa6b875415d7e0686a97a81b45d27a9213f8dd60b2730ce017d3da9ac8 not found: ID does not exist"
Nov 01 10:02:24.853419 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.853386    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"]
Nov 01 10:02:24.899677 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.899640    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfdd13-57786d88df-vs8xg"]
Nov 01 10:02:24.927235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:24.927201    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddb67ed-071b-4181-890f-56355834104f" path="/var/lib/kubelet/pods/dddb67ed-071b-4181-890f-56355834104f/volumes"
Nov 01 10:02:25.288316 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:25.288253    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:25.443615 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:25.443578    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:26.418085 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:26.418045    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:26.577451 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:26.577411    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:28.362346 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:28.362309    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:31.023178 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:31.023145    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:32.824940 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:32.824906    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:34.076721 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:34.076694    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-h6b62_f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad/kuadrant-console-plugin/0.log"
Nov 01 10:02:35.042817 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:35.042779    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"]
Nov 01 10:02:35.043080 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:35.043047    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy" containerID="cri-o://71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4" gracePeriod=30
Nov 01 10:02:35.620699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:35.620666    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:35.762140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:35.762108    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-l9ctz_f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e/manager/0.log"
Nov 01 10:02:35.840901 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:35.840840    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:38.601999 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:38.601967    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8jqs6_9f09aaaf-f882-4e11-b764-bf87a9351d3e/manager/0.log"
Nov 01 10:02:40.288841 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.288812    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 10:02:40.378359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378286    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378332    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378374    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378405    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378432    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378500    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378575    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378631    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378662    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj\") pod \"7f19af84-35e1-4ea1-97d9-68396e7099f1\" (UID: \"7f19af84-35e1-4ea1-97d9-68396e7099f1\") "
Nov 01 10:02:40.378984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378574    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:40.378984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378711    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:40.378984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378779    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:40.378984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378904    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:40.379348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.378981    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:02:40.379348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.379137    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f19af84-35e1-4ea1-97d9-68396e7099f1-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.379348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.379159    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.379348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.379175    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.379348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.379189    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.379348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.379202    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.380740 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.380715    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "kube-api-access-6p6xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:40.380859 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.380720    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:02:40.380859 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.380767    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:40.380957 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.380856    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy" pod "7f19af84-35e1-4ea1-97d9-68396e7099f1" (UID: "7f19af84-35e1-4ea1-97d9-68396e7099f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:40.479976 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.479941    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p6xj\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-kube-api-access-6p6xj\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.479976 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.479972    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.480181 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.479984    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.480181 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.479996    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f19af84-35e1-4ea1-97d9-68396e7099f1-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:40.785292 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.785263    4314 generic.go:356] "Generic (PLEG): container finished" podID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerID="71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4" exitCode=0
Nov 01 10:02:40.785481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.785308    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"
Nov 01 10:02:40.785481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.785352    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" event={"ID":"7f19af84-35e1-4ea1-97d9-68396e7099f1","Type":"ContainerDied","Data":"71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4"}
Nov 01 10:02:40.785481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.785408    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8" event={"ID":"7f19af84-35e1-4ea1-97d9-68396e7099f1","Type":"ContainerDied","Data":"35fb55ce6a4c5ede93a56bfad6da6ad9d9daa35b46ecead0fff925da743b2fb1"}
Nov 01 10:02:40.785481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.785431    4314 scope.go:122] "RemoveContainer" containerID="71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4"
Nov 01 10:02:40.795315 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.795298    4314 scope.go:122] "RemoveContainer" containerID="71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4"
Nov 01 10:02:40.795607 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:02:40.795589    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4\": container with ID starting with 71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4 not found: ID does not exist" containerID="71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4"
Nov 01 10:02:40.795689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.795619    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4"} err="failed to get container status \"71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4\": rpc error: code = NotFound desc = could not find container \"71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4\": container with ID starting with 71bcb12a81a88a3b66521cb2019e1378caedd3be0935a8a63f6ca290eabb17c4 not found: ID does not exist"
Nov 01 10:02:40.941194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.941155    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"]
Nov 01 10:02:40.987425 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:40.987397    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bn6q8"]
Nov 01 10:02:41.321512 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:02:41.321472    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:02:42.927070 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:42.927024    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" path="/var/lib/kubelet/pods/7f19af84-35e1-4ea1-97d9-68396e7099f1/volumes"
Nov 01 10:02:44.508753 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:44.508712    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:45.001699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.001665    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"]
Nov 01 10:02:45.002017 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002004    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy"
Nov 01 10:02:45.002017 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002018    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002036    4314 cpu_manager.go:414] "containerMap: removing container" podUID="cbc11a28-54a9-4e95-a393-533cdff0d576" containerName="revoke-keys"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002042    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc11a28-54a9-4e95-a393-533cdff0d576" containerName="revoke-keys"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002060    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" containerName="payload-pre-processing"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002065    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" containerName="payload-pre-processing"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002076    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dddb67ed-071b-4181-890f-56355834104f" containerName="payload-processing"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002082    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb67ed-071b-4181-890f-56355834104f" containerName="payload-processing"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002134    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2757fbbc-4757-41f2-b7b9-da1397533569" containerName="payload-pre-processing"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002143    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb67ed-071b-4181-890f-56355834104f" containerName="payload-processing"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002152    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f19af84-35e1-4ea1-97d9-68396e7099f1" containerName="istio-proxy"
Nov 01 10:02:45.002161 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.002159    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc11a28-54a9-4e95-a393-533cdff0d576" containerName="revoke-keys"
Nov 01 10:02:45.006996 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.006979    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.056180 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.056137    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7e1b7946-openshift-default-dockercfg-kbvgg\""
Nov 01 10:02:45.078746 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.078710    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 10:02:45.090648 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.090607    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:45.126214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126177    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126389 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126238    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126457 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126431    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126561 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126463    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126561 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126544    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126688 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126599    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126688 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126663    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2v5\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126848 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126822    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.126933 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.126882    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.127180 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.127154    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"]
Nov 01 10:02:45.227771 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.227729    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.227771 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.227779    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.227900    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.227943    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.227980    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228027    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228269 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228168    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2v5\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228269 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228227    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228402 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228281    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228402 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228357    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228410    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228502    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.228668 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.228547    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.229042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.229023    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.230303 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.230279    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.230983 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.230961    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.267714 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.267633    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:45.285786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.285756    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.286168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.286149    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2v5\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5\") pod \"e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") " pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.320072 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.320044    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:45.434874 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.434797    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:45.736357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.736334    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"]
Nov 01 10:02:45.738435 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:02:45.738399    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9de6c4_d3b8_4bde_84d9_8a31f7730cc8.slice/crio-a7cd97d499e5a0dc30bc56bd9f8da552a383b1bad4075a4884ec230041c76a45 WatchSource:0}: Error finding container a7cd97d499e5a0dc30bc56bd9f8da552a383b1bad4075a4884ec230041c76a45: Status 404 returned error can't find the container with id a7cd97d499e5a0dc30bc56bd9f8da552a383b1bad4075a4884ec230041c76a45
Nov 01 10:02:45.740573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.740518    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:02:45.740686 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.740614    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:02:45.740686 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.740660    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:02:45.815178 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.815145    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" event={"ID":"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8","Type":"ContainerStarted","Data":"1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"}
Nov 01 10:02:45.815331 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.815212    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" event={"ID":"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8","Type":"ContainerStarted","Data":"a7cd97d499e5a0dc30bc56bd9f8da552a383b1bad4075a4884ec230041c76a45"}
Nov 01 10:02:45.929727 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:45.929678    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podStartSLOduration=1.929663709 podStartE2EDuration="1.929663709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:02:45.928389681 +0000 UTC m=+5439.122784237" watchObservedRunningTime="2026-11-01 10:02:45.929663709 +0000 UTC m=+5439.124058264"
Nov 01 10:02:46.320798 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:46.320753    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:46.325882 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:46.325854    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:46.820871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:46.820841    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:46.821821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:46.821800    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:02:48.068689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:48.068655    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:48.270381 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:48.270345    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:49.047164 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:49.047088    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"]
Nov 01 10:02:49.093759 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:49.093725    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97cdac72-rw9sk"]
Nov 01 10:02:50.926641 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:50.926610    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00204c7c-c7b6-408f-b8e6-4426cecd2cee" path="/var/lib/kubelet/pods/00204c7c-c7b6-408f-b8e6-4426cecd2cee/volumes"
Nov 01 10:02:51.367977 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.367938    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:51.527675 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.527641    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:51.763986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.763956    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"]
Nov 01 10:02:51.767749 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.767729    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:51.821666 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.821641    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"]
Nov 01 10:02:51.864841 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.864811    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7e1b7946\""
Nov 01 10:02:51.893552 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.893504    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:51.893744 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.893723    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjfx\" (UniqueName: \"kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx\") pod \"payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:51.911526 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.911498    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"]
Nov 01 10:02:51.915201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.915179    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:51.966938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.966911    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7e1b7946-dockercfg-mw7h5\""
Nov 01 10:02:51.969985 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.969959    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"]
Nov 01 10:02:51.995085 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.995046    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjfx\" (UniqueName: \"kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx\") pod \"payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:51.995262 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.995103    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:51.995262 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.995168    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtq2\" (UniqueName: \"kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2\") pod \"payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:51.995408 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.995288    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:51.995843 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:51.995823    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:52.053475 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.053399    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjfx\" (UniqueName: \"kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx\") pod \"payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:52.081259 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.081231    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:52.096480 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.096448    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:52.096606 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.096591    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtq2\" (UniqueName: \"kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2\") pod \"payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:52.097544 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.097510    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:52.154585 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.154526    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtq2\" (UniqueName: \"kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2\") pod \"payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:52.227701 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.227655    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:52.446326 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.446301    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"]
Nov 01 10:02:52.448303 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:02:52.448263    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb953bf51_cec8_4c3a_b984_ecef3c505d8d.slice/crio-192dc2cb5e708779c38568cb26f0992c8777bd966b3c888c47e6d17b6f6de479 WatchSource:0}: Error finding container 192dc2cb5e708779c38568cb26f0992c8777bd966b3c888c47e6d17b6f6de479: Status 404 returned error can't find the container with id 192dc2cb5e708779c38568cb26f0992c8777bd966b3c888c47e6d17b6f6de479
Nov 01 10:02:52.596249 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.596220    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"]
Nov 01 10:02:52.598937 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:02:52.598911    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba03296_c97f_46e4_8dd7_d3b0ec18ee4f.slice/crio-e728806e11847ae915285d0c0e0e4105794a2631c4ec4689798668efc61c7b48 WatchSource:0}: Error finding container e728806e11847ae915285d0c0e0e4105794a2631c4ec4689798668efc61c7b48: Status 404 returned error can't find the container with id e728806e11847ae915285d0c0e0e4105794a2631c4ec4689798668efc61c7b48
Nov 01 10:02:52.853361 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.853328    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf" event={"ID":"b953bf51-cec8-4c3a-b984-ecef3c505d8d","Type":"ContainerStarted","Data":"f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5"}
Nov 01 10:02:52.853600 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.853379    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf" event={"ID":"b953bf51-cec8-4c3a-b984-ecef3c505d8d","Type":"ContainerStarted","Data":"192dc2cb5e708779c38568cb26f0992c8777bd966b3c888c47e6d17b6f6de479"}
Nov 01 10:02:52.853600 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.853425    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:02:52.855104 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.855068    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6" event={"ID":"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f","Type":"ContainerStarted","Data":"9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2"}
Nov 01 10:02:52.855240 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.855133    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:02:52.855240 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.855158    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6" event={"ID":"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f","Type":"ContainerStarted","Data":"e728806e11847ae915285d0c0e0e4105794a2631c4ec4689798668efc61c7b48"}
Nov 01 10:02:52.964389 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:52.964332    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf" podStartSLOduration=1.964317866 podStartE2EDuration="1.964317866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:02:52.961961573 +0000 UTC m=+5446.156356128" watchObservedRunningTime="2026-11-01 10:02:52.964317866 +0000 UTC m=+5446.158712421"
Nov 01 10:02:53.072806 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:53.072752    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6" podStartSLOduration=2.072738243 podStartE2EDuration="2.072738243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:02:53.070236332 +0000 UTC m=+5446.264630950" watchObservedRunningTime="2026-11-01 10:02:53.072738243 +0000 UTC m=+5446.267132808"
Nov 01 10:02:54.631886 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:54.631849    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:54.807237 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:54.807197    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:55.039672 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:55.039637    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:57.256658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:57.256622    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:02:59.779148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:02:59.779112    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:00.058430 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:00.058337    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"]
Nov 01 10:03:00.104359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:00.104321    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13753ffb-nf988"]
Nov 01 10:03:00.927151 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:00.927109    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192ec866-1ee6-437a-a406-ea4e5b73deae" path="/var/lib/kubelet/pods/192ec866-1ee6-437a-a406-ea4e5b73deae/volumes"
Nov 01 10:03:03.883553 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:03.883496    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:03:03.884074 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:03.883595    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:03:04.694596 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:04.694565    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:08.054297 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:08.054262    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"]
Nov 01 10:03:08.099545 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:08.099506    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12a95ba-gqcnn"]
Nov 01 10:03:08.926992 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:08.926956    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19fde78-6660-41c5-84f2-21675acca153" path="/var/lib/kubelet/pods/a19fde78-6660-41c5-84f2-21675acca153/volumes"
Nov 01 10:03:09.619784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:09.619749    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:11.323168 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:03:11.323133    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:03:11.953140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:11.953051    4314 generic.go:356] "Generic (PLEG): container finished" podID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerID="1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8" exitCode=0
Nov 01 10:03:11.953140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:11.953118    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" event={"ID":"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8","Type":"ContainerDied","Data":"1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"}
Nov 01 10:03:11.953510 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:11.953495    4314 scope.go:122] "RemoveContainer" containerID="1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"
Nov 01 10:03:11.954756 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:11.954722    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:03:11.954871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:11.954794    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:03:11.954871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:11.954830    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:03:12.990011 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:12.989962    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" event={"ID":"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8","Type":"ContainerStarted","Data":"0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df"}
Nov 01 10:03:13.320436 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:13.320344    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:03:13.321818 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:13.321788    4314 patch_prober.go:29] interesting pod/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 01 10:03:13.321958 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:13.321843    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 01 10:03:14.321049 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:14.321010    4314 patch_prober.go:29] interesting pod/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 01 10:03:14.321481 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:14.321074    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 01 10:03:14.854843 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:14.854807    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:15.320766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:15.320727    4314 patch_prober.go:29] interesting pod/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 01 10:03:15.320971 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:15.320793    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 01 10:03:16.320595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:16.320556    4314 patch_prober.go:29] interesting pod/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 01 10:03:16.321105 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:16.320636    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 01 10:03:16.821276 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:16.821227    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:03:17.321247 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:17.321208    4314 patch_prober.go:29] interesting pod/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 01 10:03:17.321692 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:17.321266    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 01 10:03:18.324187 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:18.324158    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:03:18.325284 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:18.325258    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:03:19.833398 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:19.833359    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:22.054628 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:22.054593    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"]
Nov 01 10:03:22.100694 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:22.100652    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19beba3f-n9p46"]
Nov 01 10:03:22.926521 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:22.926484    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49bdc99-fdc6-40c6-8027-43d0616652b5" path="/var/lib/kubelet/pods/a49bdc99-fdc6-40c6-8027-43d0616652b5/volumes"
Nov 01 10:03:24.563627 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.563591    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"]
Nov 01 10:03:24.565871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.565853    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.605624 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.605595    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vhb\" (UniqueName: \"kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.605788 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.605648    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.605788 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.605674    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.663392 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.663365    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:03:24.663576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.663417    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:03:24.663576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.663498    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:03:24.663681 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.663609    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:03:24.665806 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.665785    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"]
Nov 01 10:03:24.706842 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.706817    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vhb\" (UniqueName: \"kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.706987 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.706866    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.706987 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.706892    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.707553 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.707508    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.709232 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.709212    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.764304 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.764267    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vhb\" (UniqueName: \"kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb\") pod \"maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:24.796821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.796794    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:24.877200 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:24.877129    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:25.240031 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:25.240007    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"]
Nov 01 10:03:25.241909 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:03:25.241885    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03d35c0_84a2_46e5_9daf_cf04126c120f.slice/crio-cb2ae6707960fd8696ee21d278592f10ed09ad0bb6422eee1d593559f57bec27 WatchSource:0}: Error finding container cb2ae6707960fd8696ee21d278592f10ed09ad0bb6422eee1d593559f57bec27: Status 404 returned error can't find the container with id cb2ae6707960fd8696ee21d278592f10ed09ad0bb6422eee1d593559f57bec27
Nov 01 10:03:26.059969 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:26.059935    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb" event={"ID":"d03d35c0-84a2-46e5-9daf-cf04126c120f","Type":"ContainerStarted","Data":"68d364e141f9d819fd53fa1de79225abbee721420698c4b2518c7eb418451b2b"}
Nov 01 10:03:26.060352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:26.059994    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb" event={"ID":"d03d35c0-84a2-46e5-9daf-cf04126c120f","Type":"ContainerStarted","Data":"cb2ae6707960fd8696ee21d278592f10ed09ad0bb6422eee1d593559f57bec27"}
Nov 01 10:03:26.215463 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:26.215406    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb" podStartSLOduration=2.215390544 podStartE2EDuration="2.215390544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:03:26.214765455 +0000 UTC m=+5479.409160011" watchObservedRunningTime="2026-11-01 10:03:26.215390544 +0000 UTC m=+5479.409785099"
Nov 01 10:03:27.067089 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:27.067052    4314 generic.go:356] "Generic (PLEG): container finished" podID="d03d35c0-84a2-46e5-9daf-cf04126c120f" containerID="68d364e141f9d819fd53fa1de79225abbee721420698c4b2518c7eb418451b2b" exitCode=0
Nov 01 10:03:27.067573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:27.067100    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb" event={"ID":"d03d35c0-84a2-46e5-9daf-cf04126c120f","Type":"ContainerDied","Data":"68d364e141f9d819fd53fa1de79225abbee721420698c4b2518c7eb418451b2b"}
Nov 01 10:03:28.204155 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.204133    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:28.340052 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.339974    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb\" (UniqueName: \"kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb\") pod \"d03d35c0-84a2-46e5-9daf-cf04126c120f\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") "
Nov 01 10:03:28.340052 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.340017    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca\") pod \"d03d35c0-84a2-46e5-9daf-cf04126c120f\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") "
Nov 01 10:03:28.340276 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.340077    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca\") pod \"d03d35c0-84a2-46e5-9daf-cf04126c120f\" (UID: \"d03d35c0-84a2-46e5-9daf-cf04126c120f\") "
Nov 01 10:03:28.340547 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.340505    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca" pod "d03d35c0-84a2-46e5-9daf-cf04126c120f" (UID: "d03d35c0-84a2-46e5-9daf-cf04126c120f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:03:28.342235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.342208    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca" pod "d03d35c0-84a2-46e5-9daf-cf04126c120f" (UID: "d03d35c0-84a2-46e5-9daf-cf04126c120f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:03:28.342336 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.342242    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb" pod "d03d35c0-84a2-46e5-9daf-cf04126c120f" (UID: "d03d35c0-84a2-46e5-9daf-cf04126c120f"). InnerVolumeSpecName "kube-api-access-27vhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:03:28.441472 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.441442    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:28.441472 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.441468    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03d35c0-84a2-46e5-9daf-cf04126c120f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:28.441472 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:28.441478    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vhb\" (UniqueName: \"kubernetes.io/projected/d03d35c0-84a2-46e5-9daf-cf04126c120f-kube-api-access-27vhb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:29.078880 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:29.078851    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"
Nov 01 10:03:29.078880 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:29.078882    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb" event={"ID":"d03d35c0-84a2-46e5-9daf-cf04126c120f","Type":"ContainerDied","Data":"cb2ae6707960fd8696ee21d278592f10ed09ad0bb6422eee1d593559f57bec27"}
Nov 01 10:03:29.079108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:29.078929    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2ae6707960fd8696ee21d278592f10ed09ad0bb6422eee1d593559f57bec27"
Nov 01 10:03:29.693743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:29.693713    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:31.359688 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:31.359653    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:34.578555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:34.578504    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:36.068383 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:36.068350    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"]
Nov 01 10:03:36.177408 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:36.177375    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"]
Nov 01 10:03:36.444305 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:36.444209    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:36.600582 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:36.600545    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:39.292853 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:39.292820    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:41.068884 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.068845    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6" podUID="0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" containerName="payload-processing" containerID="cri-o://9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2" gracePeriod=25
Nov 01 10:03:41.178081 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.178036    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf" podUID="b953bf51-cec8-4c3a-b984-ecef3c505d8d" containerName="payload-pre-processing" containerID="cri-o://f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5" gracePeriod=25
Nov 01 10:03:41.323182 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:03:41.323106    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:03:41.325733 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.325705    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:03:41.438209 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.438176    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:03:41.464945 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.464918    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume\") pod \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") "
Nov 01 10:03:41.465128 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.465003    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2\" (UniqueName: \"kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2\") pod \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\" (UID: \"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f\") "
Nov 01 10:03:41.465359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.465332    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume" pod "0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" (UID: "0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:03:41.467118 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.467090    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2" pod "0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" (UID: "0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f"). InnerVolumeSpecName "kube-api-access-pwtq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:03:41.565648 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.565618    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx\" (UniqueName: \"kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx\") pod \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") "
Nov 01 10:03:41.565840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.565679    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume\") pod \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\" (UID: \"b953bf51-cec8-4c3a-b984-ecef3c505d8d\") "
Nov 01 10:03:41.565985 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.565957    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:41.565985 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.565978    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtq2\" (UniqueName: \"kubernetes.io/projected/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f-kube-api-access-pwtq2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:41.566077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.566042    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume" pod "b953bf51-cec8-4c3a-b984-ecef3c505d8d" (UID: "b953bf51-cec8-4c3a-b984-ecef3c505d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:03:41.567829 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.567803    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx" pod "b953bf51-cec8-4c3a-b984-ecef3c505d8d" (UID: "b953bf51-cec8-4c3a-b984-ecef3c505d8d"). InnerVolumeSpecName "kube-api-access-lkjfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:03:41.667184 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.667100    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjfx\" (UniqueName: \"kubernetes.io/projected/b953bf51-cec8-4c3a-b984-ecef3c505d8d-kube-api-access-lkjfx\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:41.667184 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:41.667131    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b953bf51-cec8-4c3a-b984-ecef3c505d8d-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:42.146786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.146754    4314 generic.go:356] "Generic (PLEG): container finished" podID="b953bf51-cec8-4c3a-b984-ecef3c505d8d" containerID="f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5" exitCode=0
Nov 01 10:03:42.147264 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.146805    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"
Nov 01 10:03:42.147264 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.146833    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf" event={"ID":"b953bf51-cec8-4c3a-b984-ecef3c505d8d","Type":"ContainerDied","Data":"f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5"}
Nov 01 10:03:42.147264 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.146890    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf" event={"ID":"b953bf51-cec8-4c3a-b984-ecef3c505d8d","Type":"ContainerDied","Data":"192dc2cb5e708779c38568cb26f0992c8777bd966b3c888c47e6d17b6f6de479"}
Nov 01 10:03:42.147264 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.146916    4314 scope.go:122] "RemoveContainer" containerID="f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5"
Nov 01 10:03:42.148289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.148262    4314 generic.go:356] "Generic (PLEG): container finished" podID="0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" containerID="9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2" exitCode=0
Nov 01 10:03:42.148391 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.148322    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"
Nov 01 10:03:42.148391 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.148350    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6" event={"ID":"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f","Type":"ContainerDied","Data":"9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2"}
Nov 01 10:03:42.148520 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.148414    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6" event={"ID":"0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f","Type":"ContainerDied","Data":"e728806e11847ae915285d0c0e0e4105794a2631c4ec4689798668efc61c7b48"}
Nov 01 10:03:42.157332 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.157284    4314 scope.go:122] "RemoveContainer" containerID="f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5"
Nov 01 10:03:42.157657 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:03:42.157634    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5\": container with ID starting with f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5 not found: ID does not exist" containerID="f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5"
Nov 01 10:03:42.157767 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.157665    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5"} err="failed to get container status \"f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5\": rpc error: code = NotFound desc = could not find container \"f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5\": container with ID starting with f93ad665c1c2f12ee1792cd3abf91777f429bda42a3bc7e68159a2b39cdffcd5 not found: ID does not exist"
Nov 01 10:03:42.157767 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.157687    4314 scope.go:122] "RemoveContainer" containerID="9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2"
Nov 01 10:03:42.166749 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.166731    4314 scope.go:122] "RemoveContainer" containerID="9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2"
Nov 01 10:03:42.167058 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:03:42.167035    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2\": container with ID starting with 9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2 not found: ID does not exist" containerID="9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2"
Nov 01 10:03:42.167146 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.167068    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2"} err="failed to get container status \"9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2\": rpc error: code = NotFound desc = could not find container \"9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2\": container with ID starting with 9b86b7ac3fa959abbf65e2ddc52835618a9aa88c0db6fb6418c97bb0bd9664d2 not found: ID does not exist"
Nov 01 10:03:42.347828 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.347745    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"]
Nov 01 10:03:42.393810 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.393775    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1b7946-696df66d69-cp6t6"]
Nov 01 10:03:42.458337 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.458303    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"]
Nov 01 10:03:42.462059 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.462028    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1b7946-5758969f9-r8dhf"]
Nov 01 10:03:42.927429 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.927398    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" path="/var/lib/kubelet/pods/0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f/volumes"
Nov 01 10:03:42.927795 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:42.927781    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b953bf51-cec8-4c3a-b984-ecef3c505d8d" path="/var/lib/kubelet/pods/b953bf51-cec8-4c3a-b984-ecef3c505d8d/volumes"
Nov 01 10:03:43.577973 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:43.577934    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:45.098190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:45.098155    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:45.376249 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:45.376162    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:45.589387 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:45.589346    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"]
Nov 01 10:03:45.635361 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:45.635271    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1b7946-wx5zb"]
Nov 01 10:03:46.927800 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:46.927768    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03d35c0-84a2-46e5-9daf-cf04126c120f" path="/var/lib/kubelet/pods/d03d35c0-84a2-46e5-9daf-cf04126c120f/volumes"
Nov 01 10:03:49.865392 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:49.865358    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:50.049174 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:50.049129    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:50.760195 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:50.760162    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"]
Nov 01 10:03:50.760423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:50.760402    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy" containerID="cri-o://0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df" gracePeriod=30
Nov 01 10:03:50.785295 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:50.785264    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:03:56.008371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.008343    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:03:56.101145 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101068    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101145 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101121    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101145 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101146    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101173    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101203    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101239    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101293    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101347    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101384    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data\") pod \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\" (UID: \"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8\") "
Nov 01 10:03:56.101802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101481    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:03:56.101802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101577    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:03:56.101802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101699    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:03:56.101802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101762    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.101802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101784    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.101802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101791    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:03:56.102152 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.101982    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:03:56.103410 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.103385    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "kube-api-access-xt2v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:03:56.103542 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.103500    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:03:56.103542 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.103510    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:03:56.103638 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.103555    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy" pod "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" (UID: "2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:03:56.202258 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202209    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.202258 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202250    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.202258 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202259    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.202258 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202270    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.202607 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202279    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.202607 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202287    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.202607 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.202296    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2v5\" (UniqueName: \"kubernetes.io/projected/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8-kube-api-access-xt2v5\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:56.223178 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.223141    4314 generic.go:356] "Generic (PLEG): container finished" podID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerID="0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df" exitCode=0
Nov 01 10:03:56.223334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.223199    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"
Nov 01 10:03:56.223334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.223229    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" event={"ID":"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8","Type":"ContainerDied","Data":"0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df"}
Nov 01 10:03:56.223334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.223287    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m" event={"ID":"2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8","Type":"ContainerDied","Data":"a7cd97d499e5a0dc30bc56bd9f8da552a383b1bad4075a4884ec230041c76a45"}
Nov 01 10:03:56.223334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.223313    4314 scope.go:122] "RemoveContainer" containerID="0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df"
Nov 01 10:03:56.233022 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.233003    4314 scope.go:122] "RemoveContainer" containerID="1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"
Nov 01 10:03:56.241343 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.241320    4314 scope.go:122] "RemoveContainer" containerID="0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df"
Nov 01 10:03:56.241663 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:03:56.241641    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df\": container with ID starting with 0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df not found: ID does not exist" containerID="0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df"
Nov 01 10:03:56.241747 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.241679    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df"} err="failed to get container status \"0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df\": rpc error: code = NotFound desc = could not find container \"0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df\": container with ID starting with 0d1acde185d8b51408ba44948021135654796f2a5b6c1f87ffb56c217cd201df not found: ID does not exist"
Nov 01 10:03:56.241747 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.241708    4314 scope.go:122] "RemoveContainer" containerID="1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"
Nov 01 10:03:56.241977 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:03:56.241957    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8\": container with ID starting with 1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8 not found: ID does not exist" containerID="1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"
Nov 01 10:03:56.242019 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.241986    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8"} err="failed to get container status \"1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8\": rpc error: code = NotFound desc = could not find container \"1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8\": container with ID starting with 1dfb897b50ca1bf5b5620f0bbdab76e0210e9f2faf6ddb582c8113bb236ad9f8 not found: ID does not exist"
Nov 01 10:03:56.378321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.378234    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"]
Nov 01 10:03:56.385091 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.385055    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7e1b7946-openshift-default-7dd484bd9f-rcj5m"]
Nov 01 10:03:56.777295 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777259    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-tlvmd"]
Nov 01 10:03:56.777648 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777635    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b953bf51-cec8-4c3a-b984-ecef3c505d8d" containerName="payload-pre-processing"
Nov 01 10:03:56.777699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777649    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b953bf51-cec8-4c3a-b984-ecef3c505d8d" containerName="payload-pre-processing"
Nov 01 10:03:56.777699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777660    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d03d35c0-84a2-46e5-9daf-cf04126c120f" containerName="revoke-keys"
Nov 01 10:03:56.777699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777666    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03d35c0-84a2-46e5-9daf-cf04126c120f" containerName="revoke-keys"
Nov 01 10:03:56.777699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777674    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" containerName="payload-processing"
Nov 01 10:03:56.777699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777679    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" containerName="payload-processing"
Nov 01 10:03:56.777699 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777700    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777705    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777713    4314 cpu_manager.go:414] "containerMap: removing container" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777718    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777772    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b953bf51-cec8-4c3a-b984-ecef3c505d8d" containerName="payload-pre-processing"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777780    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03d35c0-84a2-46e5-9daf-cf04126c120f" containerName="revoke-keys"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777786    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba03296-c97f-46e4-8dd7-d3b0ec18ee4f" containerName="payload-processing"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777794    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy"
Nov 01 10:03:56.777902 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.777800    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" containerName="istio-proxy"
Nov 01 10:03:56.781947 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.781929    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.805117 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.805088    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpz2s\" (UniqueName: \"kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.805117 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.805118    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.805293 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.805210    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.831278 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.831252    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 10:03:56.831409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.831311    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 10:03:56.831409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.831376    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ld8j5\""
Nov 01 10:03:56.831409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.831393    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 10:03:56.879525 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.879498    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-tlvmd"]
Nov 01 10:03:56.905676 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.905649    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpz2s\" (UniqueName: \"kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.905822 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.905682    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.905822 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.905735    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.908247 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.908219    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.908352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.908249    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:56.928437 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.928413    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8" path="/var/lib/kubelet/pods/2a9de6c4-d3b8-4bde-84d9-8a31f7730cc8/volumes"
Nov 01 10:03:56.963308 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:56.963285    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpz2s\" (UniqueName: \"kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s\") pod \"maas-controller-7d89f6bbc8-tlvmd\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") " pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:57.094947 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:57.094873    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:03:57.461417 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:57.461389    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-tlvmd"]
Nov 01 10:03:57.462917 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:03:57.462886    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d797f0_3885_4b83_b2d3_67b112f712c2.slice/crio-393fe1848b5e0e558b658dcfcef810e865c13e094545b0c03d7ade21cd393367 WatchSource:0}: Error finding container 393fe1848b5e0e558b658dcfcef810e865c13e094545b0c03d7ade21cd393367: Status 404 returned error can't find the container with id 393fe1848b5e0e558b658dcfcef810e865c13e094545b0c03d7ade21cd393367
Nov 01 10:03:57.464325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:57.464290    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 10:03:58.234359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:03:58.234324    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd" event={"ID":"11d797f0-3885-4b83-b2d3-67b112f712c2","Type":"ContainerStarted","Data":"393fe1848b5e0e558b658dcfcef810e865c13e094545b0c03d7ade21cd393367"}
Nov 01 10:04:01.254043 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:01.254007    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd" event={"ID":"11d797f0-3885-4b83-b2d3-67b112f712c2","Type":"ContainerStarted","Data":"54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6"}
Nov 01 10:04:01.254490 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:01.254147    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:04:01.366113 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:01.366059    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd" podStartSLOduration=1.9144487940000001 podStartE2EDuration="5.366046281s" totalImagesPullingTime="3.451597487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:01.365124491 +0000 UTC m=+5514.559519071" watchObservedRunningTime="2026-11-01 10:04:01.366046281 +0000 UTC m=+5514.560440859"
Nov 01 10:04:11.322766 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:04:11.322726    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:04:11.398220 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:11.398187    4314 scope.go:122] "RemoveContainer" containerID="92748f11719a780ffe23dcaa42dc9351fc4b83def070ab4bed6c575e3079bab1"
Nov 01 10:04:11.406703 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:11.406682    4314 scope.go:122] "RemoveContainer" containerID="0884fd5d7f53f320625d17de919ea710179965339ad1db947f98fc5e2f2f1b34"
Nov 01 10:04:11.414261 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:11.414245    4314 scope.go:122] "RemoveContainer" containerID="222d82a0278e5be4d94a975011a4afe5c9f902a0d1dce6616fe556ec911c6acf"
Nov 01 10:04:12.262408 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:12.262376    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:04:19.921627 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.921524    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"]
Nov 01 10:04:19.926925 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.926901    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:19.978071 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.978038    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:04:19.978289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.978047    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:04:19.978396 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.978074    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:04:19.978498 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.978351    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:04:19.981453 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:19.981384    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"]
Nov 01 10:04:20.017757 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.017724    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.017910 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.017854    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqdd6\" (UniqueName: \"kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.018051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.018026    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.073163 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.073130    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:20.119274 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.119239    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdd6\" (UniqueName: \"kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.119435 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.119283    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.119435 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.119326    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.119959 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.119938    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.121707 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.121683    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.177143 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.177065    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdd6\" (UniqueName: \"kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6\") pod \"maas-api-revoke-sub-85237e45-7hdcb\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.223862 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.223825    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:20.239495 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.239465    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:20.605611 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.605589    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"]
Nov 01 10:04:20.607797 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:04:20.607759    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3375511_49ce_49ce_b37b_b93e5b823c87.slice/crio-eef1ca7a682bbaf453eb105a3883b9414d2ec2d516b0e9b9dc3259d5215e6cc0 WatchSource:0}: Error finding container eef1ca7a682bbaf453eb105a3883b9414d2ec2d516b0e9b9dc3259d5215e6cc0: Status 404 returned error can't find the container with id eef1ca7a682bbaf453eb105a3883b9414d2ec2d516b0e9b9dc3259d5215e6cc0
Nov 01 10:04:20.852482 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.852448    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"]
Nov 01 10:04:20.857458 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.857405    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.906457 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.906414    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 10:04:20.906599 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.906520    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\""
Nov 01 10:04:20.926802 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.926763    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.926814    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.926917    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.926960    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.926979    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.927115    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927331 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.927148    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjkz\" (UniqueName: \"kubernetes.io/projected/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-kube-api-access-2tjkz\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927331 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.927182    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.927331 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.927272    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:20.955359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:20.955333    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"]
Nov 01 10:04:21.028736 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028699    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.028736 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028737    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029002 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028773    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029002 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028805    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029002 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028823    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029002 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028855    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029002 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028871    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjkz\" (UniqueName: \"kubernetes.io/projected/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-kube-api-access-2tjkz\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029002 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.028893    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029338 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.029094    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029338 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.029162    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029338 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.029242    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029338 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.029290    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.029486 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.029448    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.030100 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.030080    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.031116 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.031094    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.031371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.031349    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.087065 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.087028    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.087215 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.087193    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjkz\" (UniqueName: \"kubernetes.io/projected/2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6-kube-api-access-2tjkz\") pod \"maas-default-gateway-openshift-default-746ff8f49f-6qxcm\" (UID: \"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.181420 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.181345    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:21.354211 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.354166    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb" event={"ID":"a3375511-49ce-49ce-b37b-b93e5b823c87","Type":"ContainerStarted","Data":"a5002344982e9756207c1ce859e3785ee281512efabbe5ad9384e39be2730b46"}
Nov 01 10:04:21.354399 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.354231    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb" event={"ID":"a3375511-49ce-49ce-b37b-b93e5b823c87","Type":"ContainerStarted","Data":"eef1ca7a682bbaf453eb105a3883b9414d2ec2d516b0e9b9dc3259d5215e6cc0"}
Nov 01 10:04:21.464957 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.464911    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb" podStartSLOduration=2.464895456 podStartE2EDuration="2.464895456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:21.463113421 +0000 UTC m=+5534.657508019" watchObservedRunningTime="2026-11-01 10:04:21.464895456 +0000 UTC m=+5534.659290026"
Nov 01 10:04:21.619910 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.619884    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"]
Nov 01 10:04:21.621715 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:04:21.621680    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e94d8d4_6cd1_4e92_b324_ae94d3bf49b6.slice/crio-39ca837fe27ab0a6edf655229b43b4fe0514e243d7973db056f7142cf58bbd3b WatchSource:0}: Error finding container 39ca837fe27ab0a6edf655229b43b4fe0514e243d7973db056f7142cf58bbd3b: Status 404 returned error can't find the container with id 39ca837fe27ab0a6edf655229b43b4fe0514e243d7973db056f7142cf58bbd3b
Nov 01 10:04:21.624141 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.624104    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:21.624211 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.624194    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:21.624252 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:21.624240    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:22.389424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:22.389387    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" event={"ID":"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6","Type":"ContainerStarted","Data":"43b39a16b7aee6876230819c0242a034495e9540c0218a7510a030f4c30b144f"}
Nov 01 10:04:22.389983 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:22.389451    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" event={"ID":"2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6","Type":"ContainerStarted","Data":"39ca837fe27ab0a6edf655229b43b4fe0514e243d7973db056f7142cf58bbd3b"}
Nov 01 10:04:22.390921 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:22.390898    4314 generic.go:356] "Generic (PLEG): container finished" podID="a3375511-49ce-49ce-b37b-b93e5b823c87" containerID="a5002344982e9756207c1ce859e3785ee281512efabbe5ad9384e39be2730b46" exitCode=0
Nov 01 10:04:22.391036 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:22.390977    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb" event={"ID":"a3375511-49ce-49ce-b37b-b93e5b823c87","Type":"ContainerDied","Data":"a5002344982e9756207c1ce859e3785ee281512efabbe5ad9384e39be2730b46"}
Nov 01 10:04:22.503257 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:22.503202    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podStartSLOduration=2.5031877270000003 podStartE2EDuration="2.503187727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:22.502057531 +0000 UTC m=+5535.696452089" watchObservedRunningTime="2026-11-01 10:04:22.503187727 +0000 UTC m=+5535.697582281"
Nov 01 10:04:23.182759 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.182718    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:23.184275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.184246    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:23.184420 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.184306    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:23.536317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.536287    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:23.653392 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.653347    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6\" (UniqueName: \"kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6\") pod \"a3375511-49ce-49ce-b37b-b93e5b823c87\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") "
Nov 01 10:04:23.653622 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.653446    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca\") pod \"a3375511-49ce-49ce-b37b-b93e5b823c87\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") "
Nov 01 10:04:23.653622 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.653561    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca\") pod \"a3375511-49ce-49ce-b37b-b93e5b823c87\" (UID: \"a3375511-49ce-49ce-b37b-b93e5b823c87\") "
Nov 01 10:04:23.653885 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.653844    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca" pod "a3375511-49ce-49ce-b37b-b93e5b823c87" (UID: "a3375511-49ce-49ce-b37b-b93e5b823c87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:04:23.655660 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.655634    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6" pod "a3375511-49ce-49ce-b37b-b93e5b823c87" (UID: "a3375511-49ce-49ce-b37b-b93e5b823c87"). InnerVolumeSpecName "kube-api-access-sqdd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:04:23.655896 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.655867    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca" pod "a3375511-49ce-49ce-b37b-b93e5b823c87" (UID: "a3375511-49ce-49ce-b37b-b93e5b823c87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:04:23.754557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.754501    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:04:23.754722 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.754559    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqdd6\" (UniqueName: \"kubernetes.io/projected/a3375511-49ce-49ce-b37b-b93e5b823c87-kube-api-access-sqdd6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:04:23.754722 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:23.754577    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3375511-49ce-49ce-b37b-b93e5b823c87-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:04:24.183061 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:24.182965    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:24.183061 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:24.183026    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:24.403439 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:24.403411    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"
Nov 01 10:04:24.403439 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:24.403434    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb" event={"ID":"a3375511-49ce-49ce-b37b-b93e5b823c87","Type":"ContainerDied","Data":"eef1ca7a682bbaf453eb105a3883b9414d2ec2d516b0e9b9dc3259d5215e6cc0"}
Nov 01 10:04:24.403732 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:24.403480    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eef1ca7a682bbaf453eb105a3883b9414d2ec2d516b0e9b9dc3259d5215e6cc0"
Nov 01 10:04:25.183000 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:25.182964    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:25.183504 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:25.183031    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:26.182068 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:26.182023    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:26.182409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:26.182376    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:27.183056 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:27.183014    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:27.183466 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:27.183085    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:28.181782 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:28.181741    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:28.182062 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:28.181809    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:29.182051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:29.182016    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Nov 01 10:04:29.182499 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:29.182071    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Nov 01 10:04:29.202866 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:29.202827    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:29.358365 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:29.358330    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:31.182494 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:31.182448    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:04:31.182998 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:31.182580    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:04:31.700958 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:31.700921    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:32.223736 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.223697    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:04:32.224261 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.223777    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:04:32.391042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.391003    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:32.523610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.523563    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-tlvmd"]
Nov 01 10:04:32.523844 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.523777    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd" podUID="11d797f0-3885-4b83-b2d3-67b112f712c2" containerName="manager" containerID="cri-o://54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6" gracePeriod=30
Nov 01 10:04:32.774233 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.774166    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:04:32.855402 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.855362    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs\") pod \"11d797f0-3885-4b83-b2d3-67b112f712c2\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") "
Nov 01 10:04:32.855402 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.855409    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s\" (UniqueName: \"kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s\") pod \"11d797f0-3885-4b83-b2d3-67b112f712c2\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") "
Nov 01 10:04:32.855732 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.855462    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert\") pod \"11d797f0-3885-4b83-b2d3-67b112f712c2\" (UID: \"11d797f0-3885-4b83-b2d3-67b112f712c2\") "
Nov 01 10:04:32.857731 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.857684    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs" pod "11d797f0-3885-4b83-b2d3-67b112f712c2" (UID: "11d797f0-3885-4b83-b2d3-67b112f712c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:04:32.857731 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.857688    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s" pod "11d797f0-3885-4b83-b2d3-67b112f712c2" (UID: "11d797f0-3885-4b83-b2d3-67b112f712c2"). InnerVolumeSpecName "kube-api-access-zpz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:04:32.857731 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.857699    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert" pod "11d797f0-3885-4b83-b2d3-67b112f712c2" (UID: "11d797f0-3885-4b83-b2d3-67b112f712c2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:04:32.956479 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.956447    4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-metrics-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:04:32.956479 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.956477    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpz2s\" (UniqueName: \"kubernetes.io/projected/11d797f0-3885-4b83-b2d3-67b112f712c2-kube-api-access-zpz2s\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:04:32.956712 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:32.956489    4314 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11d797f0-3885-4b83-b2d3-67b112f712c2-webhook-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:04:33.265755 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.265712    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:04:33.266284 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.265775    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:04:33.452004 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.451969    4314 generic.go:356] "Generic (PLEG): container finished" podID="11d797f0-3885-4b83-b2d3-67b112f712c2" containerID="54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6" exitCode=0
Nov 01 10:04:33.452237 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.452016    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd"
Nov 01 10:04:33.452237 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.452045    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd" event={"ID":"11d797f0-3885-4b83-b2d3-67b112f712c2","Type":"ContainerDied","Data":"54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6"}
Nov 01 10:04:33.452237 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.452099    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-tlvmd" event={"ID":"11d797f0-3885-4b83-b2d3-67b112f712c2","Type":"ContainerDied","Data":"393fe1848b5e0e558b658dcfcef810e865c13e094545b0c03d7ade21cd393367"}
Nov 01 10:04:33.452237 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.452130    4314 scope.go:122] "RemoveContainer" containerID="54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6"
Nov 01 10:04:33.473287 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.473262    4314 scope.go:122] "RemoveContainer" containerID="54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6"
Nov 01 10:04:33.473708 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:04:33.473684    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6\": container with ID starting with 54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6 not found: ID does not exist" containerID="54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6"
Nov 01 10:04:33.473812 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.473722    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6"} err="failed to get container status \"54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6\": rpc error: code = NotFound desc = could not find container \"54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6\": container with ID starting with 54dd4a5e847f54f0534655dbe73bfd022e820d710601ab47c503fdbccdb1ccb6 not found: ID does not exist"
Nov 01 10:04:33.609932 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.609832    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-tlvmd"]
Nov 01 10:04:33.654634 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.654593    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-tlvmd"]
Nov 01 10:04:33.750195 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.750161    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:33.750600 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.750579    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:33.751267 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:33.751248    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm"
Nov 01 10:04:34.886108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:34.886077    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:04:34.926891 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:34.926849    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d797f0-3885-4b83-b2d3-67b112f712c2" path="/var/lib/kubelet/pods/11d797f0-3885-4b83-b2d3-67b112f712c2/volumes"
Nov 01 10:04:35.039741 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.039703    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-z9hs6"]
Nov 01 10:04:35.040158 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.040141    4314 cpu_manager.go:414] "containerMap: removing container" podUID="11d797f0-3885-4b83-b2d3-67b112f712c2" containerName="manager"
Nov 01 10:04:35.040206 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.040160    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d797f0-3885-4b83-b2d3-67b112f712c2" containerName="manager"
Nov 01 10:04:35.040206 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.040188    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a3375511-49ce-49ce-b37b-b93e5b823c87" containerName="revoke-keys"
Nov 01 10:04:35.040206 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.040194    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3375511-49ce-49ce-b37b-b93e5b823c87" containerName="revoke-keys"
Nov 01 10:04:35.040322 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.040249    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d797f0-3885-4b83-b2d3-67b112f712c2" containerName="manager"
Nov 01 10:04:35.040322 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.040262    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3375511-49ce-49ce-b37b-b93e5b823c87" containerName="revoke-keys"
Nov 01 10:04:35.044360 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.044340    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.075525 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.075486    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d16903d-3097-4cab-8cdb-fe2491d19902-webhook-cert\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.075719 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.075676    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57h4\" (UniqueName: \"kubernetes.io/projected/5d16903d-3097-4cab-8cdb-fe2491d19902-kube-api-access-h57h4\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.075786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.075735    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d16903d-3097-4cab-8cdb-fe2491d19902-metrics-certs\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.093982 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.093953    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 10:04:35.140146 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.140058    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 10:04:35.140146 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.140059    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ld8j5\""
Nov 01 10:04:35.140146 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.140127    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 10:04:35.142786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.142748    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-z9hs6"]
Nov 01 10:04:35.176700 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.176665    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57h4\" (UniqueName: \"kubernetes.io/projected/5d16903d-3097-4cab-8cdb-fe2491d19902-kube-api-access-h57h4\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.176875 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.176720    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d16903d-3097-4cab-8cdb-fe2491d19902-metrics-certs\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.176875 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.176766    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d16903d-3097-4cab-8cdb-fe2491d19902-webhook-cert\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.179409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.179376    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d16903d-3097-4cab-8cdb-fe2491d19902-metrics-certs\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.179582 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.179376    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d16903d-3097-4cab-8cdb-fe2491d19902-webhook-cert\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.243658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.243623    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57h4\" (UniqueName: \"kubernetes.io/projected/5d16903d-3097-4cab-8cdb-fe2491d19902-kube-api-access-h57h4\") pod \"maas-controller-7d89f6bbc8-z9hs6\" (UID: \"5d16903d-3097-4cab-8cdb-fe2491d19902\") " pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.356665 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.356620    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:35.727470 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.727439    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-z9hs6"]
Nov 01 10:04:35.728595 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:04:35.728566    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d16903d_3097_4cab_8cdb_fe2491d19902.slice/crio-99354c0153f6b25ffe50a24705c6fdaf115c0c9e0c356cb2740f5a815a1b3af9 WatchSource:0}: Error finding container 99354c0153f6b25ffe50a24705c6fdaf115c0c9e0c356cb2740f5a815a1b3af9: Status 404 returned error can't find the container with id 99354c0153f6b25ffe50a24705c6fdaf115c0c9e0c356cb2740f5a815a1b3af9
Nov 01 10:04:35.981643 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.981613    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"]
Nov 01 10:04:35.986227 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:35.986205    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.005172 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.005144    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"]
Nov 01 10:04:36.008288 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.008271    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.044164 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.044133    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"]
Nov 01 10:04:36.083525 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083487    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3959e93c-0abf-409a-9f46-d31930d382c6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.083525 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083529    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083643    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083671    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083704    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083722    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083740    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083778    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3959e93c-0abf-409a-9f46-d31930d382c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083797    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.083835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083825    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083883    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083899    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083942    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083968    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3959e93c-0abf-409a-9f46-d31930d382c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.083990    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.084009    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vgr\" (UniqueName: \"kubernetes.io/projected/3959e93c-0abf-409a-9f46-d31930d382c6-kube-api-access-47vgr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.084028    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qpq4\" (UniqueName: \"kubernetes.io/projected/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-kube-api-access-9qpq4\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.084321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.084096    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.091926 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.091903    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"]
Nov 01 10:04:36.184991 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.184956    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3959e93c-0abf-409a-9f46-d31930d382c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185005    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185039    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185072    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185098    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185146    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185173    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3959e93c-0abf-409a-9f46-d31930d382c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185205    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185236    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vgr\" (UniqueName: \"kubernetes.io/projected/3959e93c-0abf-409a-9f46-d31930d382c6-kube-api-access-47vgr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185268    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qpq4\" (UniqueName: \"kubernetes.io/projected/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-kube-api-access-9qpq4\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185309    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185371    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3959e93c-0abf-409a-9f46-d31930d382c6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185407    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185457    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185479    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185483    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185510    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.185609 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185584    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.186248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185616    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.186248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185648    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.186248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185801    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.186248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.185857    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3959e93c-0abf-409a-9f46-d31930d382c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.186248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.186110    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.186248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.186139    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.186637 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.186478    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.187045 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.187001    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.187157 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.187055    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.187291 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.187270    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.188562 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.188506    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.188843 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.188823    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.189213 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.189193    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3959e93c-0abf-409a-9f46-d31930d382c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.189272 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.189193    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3959e93c-0abf-409a-9f46-d31930d382c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.244054 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.243963    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3959e93c-0abf-409a-9f46-d31930d382c6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.244054 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.243969    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.246011 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.245988    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qpq4\" (UniqueName: \"kubernetes.io/projected/4cc4fee6-5729-487e-9ae7-e7997f9fcb3e-kube-api-access-9qpq4\") pod \"maas-default-gateway-openshift-default-746ff8f49f-n4qzc\" (UID: \"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.250219 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.250199    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vgr\" (UniqueName: \"kubernetes.io/projected/3959e93c-0abf-409a-9f46-d31930d382c6-kube-api-access-47vgr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-h8njt\" (UID: \"3959e93c-0abf-409a-9f46-d31930d382c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.299514 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.299474    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:36.320935 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.320900    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:36.469019 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.468981    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6" event={"ID":"5d16903d-3097-4cab-8cdb-fe2491d19902","Type":"ContainerStarted","Data":"99354c0153f6b25ffe50a24705c6fdaf115c0c9e0c356cb2740f5a815a1b3af9"}
Nov 01 10:04:36.750009 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:04:36.749968    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3959e93c_0abf_409a_9f46_d31930d382c6.slice/crio-38eb7f4c37d675f93bba35a5bb1b44ae4c8d46e000ab5b80b9d6f80e580d61c8 WatchSource:0}: Error finding container 38eb7f4c37d675f93bba35a5bb1b44ae4c8d46e000ab5b80b9d6f80e580d61c8: Status 404 returned error can't find the container with id 38eb7f4c37d675f93bba35a5bb1b44ae4c8d46e000ab5b80b9d6f80e580d61c8
Nov 01 10:04:36.751129 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:04:36.751093    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc4fee6_5729_487e_9ae7_e7997f9fcb3e.slice/crio-950b5fa974e021be92531befecb243085a8018cb57f583ca7f2de9a13f42e0d6 WatchSource:0}: Error finding container 950b5fa974e021be92531befecb243085a8018cb57f583ca7f2de9a13f42e0d6: Status 404 returned error can't find the container with id 950b5fa974e021be92531befecb243085a8018cb57f583ca7f2de9a13f42e0d6
Nov 01 10:04:36.751129 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.751101    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"]
Nov 01 10:04:36.752698 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.752660    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:36.752801 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.752749    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:36.752906 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.752794    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:36.753509 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.753480    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:36.753618 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.753570    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:36.753696 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.753618    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:04:36.753927 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:36.753902    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"]
Nov 01 10:04:37.049844 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.049809    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"]
Nov 01 10:04:37.054583 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.054525    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0230f1eb-dhr5z"]
Nov 01 10:04:37.494557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.492582    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" event={"ID":"3959e93c-0abf-409a-9f46-d31930d382c6","Type":"ContainerStarted","Data":"9c149454810e90a08acae6dbeefc5ebb4ebbdfe1eb294fef1a3f59c38523f8f7"}
Nov 01 10:04:37.494557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.492661    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" event={"ID":"3959e93c-0abf-409a-9f46-d31930d382c6","Type":"ContainerStarted","Data":"38eb7f4c37d675f93bba35a5bb1b44ae4c8d46e000ab5b80b9d6f80e580d61c8"}
Nov 01 10:04:37.507562 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.506879    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" event={"ID":"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e","Type":"ContainerStarted","Data":"a2d88d469d0355559b6a0f6ceefe182ff8a7b465834c41cab9268acfd22eaee8"}
Nov 01 10:04:37.507562 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.506963    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" event={"ID":"4cc4fee6-5729-487e-9ae7-e7997f9fcb3e","Type":"ContainerStarted","Data":"950b5fa974e021be92531befecb243085a8018cb57f583ca7f2de9a13f42e0d6"}
Nov 01 10:04:37.517301 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.516445    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6" event={"ID":"5d16903d-3097-4cab-8cdb-fe2491d19902","Type":"ContainerStarted","Data":"4bed63bbed23fbb828ef92c852042dc968c9bdb532d8def1f5dd5a9cd9f3857a"}
Nov 01 10:04:37.517301 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.516570    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:37.612602 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.612454    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podStartSLOduration=2.612435564 podStartE2EDuration="2.612435564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:37.612032487 +0000 UTC m=+5550.806427044" watchObservedRunningTime="2026-11-01 10:04:37.612435564 +0000 UTC m=+5550.806830120"
Nov 01 10:04:37.739593 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.738590    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podStartSLOduration=2.738571786 podStartE2EDuration="2.738571786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:37.736958906 +0000 UTC m=+5550.931353462" watchObservedRunningTime="2026-11-01 10:04:37.738571786 +0000 UTC m=+5550.932966341"
Nov 01 10:04:37.853757 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:37.853650    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6" podStartSLOduration=2.9704458000000002 podStartE2EDuration="3.85363349s" totalImagesPullingTime="883.18769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:37.852028516 +0000 UTC m=+5551.046423073" watchObservedRunningTime="2026-11-01 10:04:37.85363349 +0000 UTC m=+5551.048028045"
Nov 01 10:04:38.303548 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.303491    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:38.304255 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.304227    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:38.304431 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.304404    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:38.322063 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.321953    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:38.322998 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.322970    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:38.323072 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.323047    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:38.927810 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:38.927774    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540ba017-f178-436c-b8b2-25987db2c84c" path="/var/lib/kubelet/pods/540ba017-f178-436c-b8b2-25987db2c84c/volumes"
Nov 01 10:04:39.302499 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:39.302015    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:39.302499 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:39.302083    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:39.321716 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:39.321681    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:39.322121 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:39.321734    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:40.300559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:40.299991    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:40.300559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:40.300058    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:40.322106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:40.322069    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:40.322519 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:40.322130    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:41.047611 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:41.047573    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"]
Nov 01 10:04:41.101554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:41.101481    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938ca959-9twpr"]
Nov 01 10:04:41.304618 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:41.303656    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:41.304618 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:41.303717    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:41.322263 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:41.322198    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:41.322708 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:41.322303    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:41.334235 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:04:41.334199    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:04:42.302017 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:42.301974    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:42.302367 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:42.302326    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:42.321760 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:42.321728    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:42.321966 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:42.321931    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:42.926983 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:42.926945    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29918d00-33ad-45c6-9885-430fe93efe2d" path="/var/lib/kubelet/pods/29918d00-33ad-45c6-9885-430fe93efe2d/volumes"
Nov 01 10:04:43.300470 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:43.300428    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:43.300710 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:43.300505    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:43.321716 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:43.321670    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:43.321882 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:43.321754    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:44.300289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:44.300246    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Nov 01 10:04:44.300768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:44.300326    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Nov 01 10:04:44.321957 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:44.321915    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 01 10:04:44.322117 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:44.321983    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 01 10:04:46.134718 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.134669    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:46.135938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.135912    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:46.136369 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.136351    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc"
Nov 01 10:04:46.363707 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.363669    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:04:46.363900 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.363744    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:04:46.405800 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.405719    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:46.406173 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.406151    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:46.406856 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:46.406840    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt"
Nov 01 10:04:48.046170 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:48.046132    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"]
Nov 01 10:04:48.092433 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:48.092395    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1fedfc-922lr"]
Nov 01 10:04:48.524130 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:48.524103    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d89f6bbc8-z9hs6"
Nov 01 10:04:48.927842 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:48.927752    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbd8357-972c-4c64-a302-9519f7bdd93f" path="/var/lib/kubelet/pods/9cbd8357-972c-4c64-a302-9519f7bdd93f/volumes"
Nov 01 10:04:52.054302 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:52.054262    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"]
Nov 01 10:04:52.100261 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:52.100224    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5887ebe1-9h4br"]
Nov 01 10:04:52.927986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:04:52.927951    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6599748-433a-4fad-81a1-a989d6df7271" path="/var/lib/kubelet/pods/d6599748-433a-4fad-81a1-a989d6df7271/volumes"
Nov 01 10:05:02.183797 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:02.183673    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:02.184558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:02.184493    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:02.448798 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:02.448689    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:02.448798 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:02.448774    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:03.096840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:03.096803    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"]
Nov 01 10:05:03.126377 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:03.126336    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794f8b20-zc25k"]
Nov 01 10:05:04.926759 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:04.926724    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f974be-0eea-4f4e-9752-4b1ff962ce6e" path="/var/lib/kubelet/pods/10f974be-0eea-4f4e-9752-4b1ff962ce6e/volumes"
Nov 01 10:05:09.911766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:09.911730    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:05:10.055093 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:10.055045    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:05:11.323343 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:05:11.323307    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:05:11.451667 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:11.451630    4314 scope.go:122] "RemoveContainer" containerID="2217efe939abe6218b4111944a69e044e91423d43637ce3e0b9af003c5b15f27"
Nov 01 10:05:21.346901 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.346864    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"]
Nov 01 10:05:21.352252 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.352233    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426486 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426439    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68734d8-e154-4730-990c-3bb7292efaa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426486 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426491    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68734d8-e154-4730-990c-3bb7292efaa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426554    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426673    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426714    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68734d8-e154-4730-990c-3bb7292efaa0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426851 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426742    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.426958 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426930    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.427007 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.426982    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klqw8\" (UniqueName: \"kubernetes.io/projected/d68734d8-e154-4730-990c-3bb7292efaa0-kube-api-access-klqw8\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.427069 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.427047    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.527788 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527755    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.527788 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527796    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klqw8\" (UniqueName: \"kubernetes.io/projected/d68734d8-e154-4730-990c-3bb7292efaa0-kube-api-access-klqw8\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527821    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527841    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68734d8-e154-4730-990c-3bb7292efaa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527861    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68734d8-e154-4730-990c-3bb7292efaa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527903    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527941    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.527978    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68734d8-e154-4730-990c-3bb7292efaa0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528435 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.528268    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528435 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.528321    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.528443    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.528511    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528733 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.528702    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.528836 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.528800    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68734d8-e154-4730-990c-3bb7292efaa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.530331 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.530305    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68734d8-e154-4730-990c-3bb7292efaa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.530524 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.530502    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68734d8-e154-4730-990c-3bb7292efaa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.552194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.552164    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"]
Nov 01 10:05:21.612598 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.612481    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klqw8\" (UniqueName: \"kubernetes.io/projected/d68734d8-e154-4730-990c-3bb7292efaa0-kube-api-access-klqw8\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.615401 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.615345    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68734d8-e154-4730-990c-3bb7292efaa0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bx5ck\" (UID: \"d68734d8-e154-4730-990c-3bb7292efaa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:21.672485 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:21.672452    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:22.113069 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:22.113033    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"]
Nov 01 10:05:22.120252 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:22.120209    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:05:22.120364 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:22.120312    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:05:22.120420 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:22.120390    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:05:22.798921 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:22.798889    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" event={"ID":"d68734d8-e154-4730-990c-3bb7292efaa0","Type":"ContainerStarted","Data":"7a46c41686bfcd640c6253a2abbd14b0d009c4c78b2ef21dd71684b113153e74"}
Nov 01 10:05:22.799313 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:22.798945    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" event={"ID":"d68734d8-e154-4730-990c-3bb7292efaa0","Type":"ContainerStarted","Data":"cd70a014b0944a9e3464b7ffe5052dd2480bc36d1afcb834d94d05591f390454"}
Nov 01 10:05:23.003791 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:23.003735    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podStartSLOduration=2.003720559 podStartE2EDuration="2.003720559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:05:22.978672724 +0000 UTC m=+5596.173067280" watchObservedRunningTime="2026-11-01 10:05:23.003720559 +0000 UTC m=+5596.198115113"
Nov 01 10:05:23.679554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:23.678605    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:23.682653 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:23.682626    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:23.682786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:23.682677    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:24.674942 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:24.674903    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:24.676155 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:24.676120    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:25.677089 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:25.677053    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:25.677801 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:25.677755    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:26.673289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:26.673249    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:26.673633 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:26.673602    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:27.674229 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:27.674174    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:27.674923 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:27.674875    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:28.673895 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:28.673850    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:28.674082 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:28.673918    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:29.673345 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:29.673306    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 01 10:05:29.673787 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:29.673366    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 01 10:05:31.674029 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:31.673987    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:31.674689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:31.674075    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:32.038068 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:32.038029    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:05:32.675093 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:32.675050    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:32.675581 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:32.675119    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:33.716726 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:33.716678    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:33.717397 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:33.716749    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:34.717142 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:34.717103    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:34.717805 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:34.717763    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:35.728557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:35.725639    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:35.728557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:35.725735    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:36.676941 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:36.676897    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:36.775736 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:36.775679    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:36.776508 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:36.776459    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:37.821558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:37.820654    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:37.821558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:37.820747    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:38.584653 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:38.584620    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:38.585839 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:05:38.585808    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck"
Nov 01 10:05:41.322637 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:05:41.322601    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:06:11.321598 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:06:11.321562    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:06:11.481088 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:11.481051    4314 scope.go:122] "RemoveContainer" containerID="b2b57f90f0837877e3a741dfe27735f33ab3d86fbf0830e8c11fe2a62f80d761"
Nov 01 10:06:11.489788 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:11.489762    4314 scope.go:122] "RemoveContainer" containerID="319e1aecd6fd8ed7d8fc896cf0115c35271815e221094c1726a626b6f9abda3b"
Nov 01 10:06:11.497739 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:11.497720    4314 scope.go:122] "RemoveContainer" containerID="ab057b57c70a42e1816fdb2be61082904a2c2e7ddb63d81721e3db72e0998294"
Nov 01 10:06:11.505402 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:11.505384    4314 scope.go:122] "RemoveContainer" containerID="93a32d56c751d3a0d6a2ccf9420e75492e5c9723cba8cd0d5a8ed94c3a48258b"
Nov 01 10:06:11.515690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:11.515664    4314 scope.go:122] "RemoveContainer" containerID="b85ef936ad95c57dbd119dd0d3bc4736c912ebe4f5df385c7fafdb362c501a49"
Nov 01 10:06:25.866309 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:25.866270    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"]
Nov 01 10:06:25.886555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:25.886509    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:25.943806 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:25.943777    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"]
Nov 01 10:06:26.046764 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.046722    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.047060 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.047035    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.047213 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.046754    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-d609fa82-kserve-self-signed-certs\""
Nov 01 10:06:26.047359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.047337    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.047573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.047545    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9gw\" (UniqueName: \"kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.047690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.047587    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.047690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.047623    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.048189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.048163    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:06:26.155616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.155090    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.155616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.155158    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.155616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.155217    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.155616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.155314    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9gw\" (UniqueName: \"kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.155616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.155347    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.155616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.155388    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.207557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.184908    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.207825 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.188012    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.208854 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.208822    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.232600 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.211999    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.234325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.234291    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.315559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.308743    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9gw\" (UniqueName: \"kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw\") pod \"e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") " pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.356467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.356423    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:26.833557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:26.832877    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:06:27.179505 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:27.179418    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"]
Nov 01 10:06:27.183405 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:06:27.183371    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fbca358_6ef0_42a2_9f39_5f5a11a9ca06.slice/crio-43372096819f591a16fab7652bba68c97db12de978cc397b78b25bff61e9cb55 WatchSource:0}: Error finding container 43372096819f591a16fab7652bba68c97db12de978cc397b78b25bff61e9cb55: Status 404 returned error can't find the container with id 43372096819f591a16fab7652bba68c97db12de978cc397b78b25bff61e9cb55
Nov 01 10:06:27.420476 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:27.420439    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc" event={"ID":"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06","Type":"ContainerStarted","Data":"3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4"}
Nov 01 10:06:27.420707 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:27.420488    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc" event={"ID":"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06","Type":"ContainerStarted","Data":"43372096819f591a16fab7652bba68c97db12de978cc397b78b25bff61e9cb55"}
Nov 01 10:06:27.531729 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:27.531672    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc" podStartSLOduration=2.531651731 podStartE2EDuration="2.531651731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:06:27.529556352 +0000 UTC m=+5660.723950902" watchObservedRunningTime="2026-11-01 10:06:27.531651731 +0000 UTC m=+5660.726046288"
Nov 01 10:06:28.368354 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:28.368320    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:30.897788 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:30.897746    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:06:32.387882 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:32.387814    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:06:34.804594 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:34.803646    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:06:34.804594 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:34.803721    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:06:36.368293 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:36.368253    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:36.381099 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:36.381066    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:36.526037 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:36.526006    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:06:41.322468 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:06:41.322433    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:06:49.047606 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:49.047570    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"]
Nov 01 10:06:49.094068 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:49.094037    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbebde98-tbbxs"]
Nov 01 10:06:50.928423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:06:50.928380    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c409c066-08cf-4e4e-aeef-ad4359c5da5e" path="/var/lib/kubelet/pods/c409c066-08cf-4e4e-aeef-ad4359c5da5e/volumes"
Nov 01 10:07:02.055610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:02.055572    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"]
Nov 01 10:07:02.101934 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:02.101909    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e171d6e-p6nhx"]
Nov 01 10:07:02.927369 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:02.927334    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc11a28-54a9-4e95-a393-533cdff0d576" path="/var/lib/kubelet/pods/cbc11a28-54a9-4e95-a393-533cdff0d576/volumes"
Nov 01 10:07:04.986700 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:04.986659    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:11.334191 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:07:11.334144    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:07:22.254469 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.254390    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"]
Nov 01 10:07:22.259136 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.259120    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.308858 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.308795    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:07:22.311923 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.311902    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"]
Nov 01 10:07:22.324167 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.324145    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpql\" (UniqueName: \"kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.324275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.324176    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.324275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.324214    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.354952 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.354929    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:07:22.354952 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.354934    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:07:22.355101 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.354971    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:07:22.425226 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.425199    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpql\" (UniqueName: \"kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.425352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.425235    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.425352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.425266    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.426035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.426012    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.427597 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.427574    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.482333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.482303    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpql\" (UniqueName: \"kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql\") pod \"maas-api-revoke-sub-38e8cb47-ntqbf\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.572903 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.572842    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:22.940346 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:22.940259    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"]
Nov 01 10:07:22.943860 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:07:22.943829    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11cbc5a_2acf_4e7e_8352_7a5cc83c6231.slice/crio-4337a0569b334b9cc803b7b06f3837516ba823bbed9a0f69fd4e378085689b7d WatchSource:0}: Error finding container 4337a0569b334b9cc803b7b06f3837516ba823bbed9a0f69fd4e378085689b7d: Status 404 returned error can't find the container with id 4337a0569b334b9cc803b7b06f3837516ba823bbed9a0f69fd4e378085689b7d
Nov 01 10:07:23.791965 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:23.791934    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf" event={"ID":"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231","Type":"ContainerStarted","Data":"dfb2c21cc5a47457d1dad7e4596985908c1871a41567d5626cdf6914d9102d47"}
Nov 01 10:07:23.792374 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:23.791984    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf" event={"ID":"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231","Type":"ContainerStarted","Data":"4337a0569b334b9cc803b7b06f3837516ba823bbed9a0f69fd4e378085689b7d"}
Nov 01 10:07:23.901991 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:23.901942    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf" podStartSLOduration=1.901928941 podStartE2EDuration="1.901928941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:07:23.900119851 +0000 UTC m=+5717.094514408" watchObservedRunningTime="2026-11-01 10:07:23.901928941 +0000 UTC m=+5717.096323519"
Nov 01 10:07:24.798359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:24.798324    4314 generic.go:356] "Generic (PLEG): container finished" podID="b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" containerID="dfb2c21cc5a47457d1dad7e4596985908c1871a41567d5626cdf6914d9102d47" exitCode=0
Nov 01 10:07:24.798776 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:24.798385    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf" event={"ID":"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231","Type":"ContainerDied","Data":"dfb2c21cc5a47457d1dad7e4596985908c1871a41567d5626cdf6914d9102d47"}
Nov 01 10:07:25.935640 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:25.935616    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:26.058557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.058477    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca\") pod \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") "
Nov 01 10:07:26.058707 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.058558    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql\" (UniqueName: \"kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql\") pod \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") "
Nov 01 10:07:26.058707 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.058638    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca\") pod \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\" (UID: \"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231\") "
Nov 01 10:07:26.058906 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.058885    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca" pod "b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" (UID: "b11cbc5a-2acf-4e7e-8352-7a5cc83c6231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:07:26.059053 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.059033    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:26.060741 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.060713    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca" pod "b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" (UID: "b11cbc5a-2acf-4e7e-8352-7a5cc83c6231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:07:26.061175 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.061155    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql" pod "b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" (UID: "b11cbc5a-2acf-4e7e-8352-7a5cc83c6231"). InnerVolumeSpecName "kube-api-access-qkpql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:07:26.160108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.160083    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpql\" (UniqueName: \"kubernetes.io/projected/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-kube-api-access-qkpql\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:26.160108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.160108    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:26.809893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.809859    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf" event={"ID":"b11cbc5a-2acf-4e7e-8352-7a5cc83c6231","Type":"ContainerDied","Data":"4337a0569b334b9cc803b7b06f3837516ba823bbed9a0f69fd4e378085689b7d"}
Nov 01 10:07:26.810076 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.809909    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4337a0569b334b9cc803b7b06f3837516ba823bbed9a0f69fd4e378085689b7d"
Nov 01 10:07:26.810076 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:26.809874    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"
Nov 01 10:07:32.728958 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:32.728926    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:35.202584 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:35.202551    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"]
Nov 01 10:07:35.227430 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:35.227399    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:35.381255 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:35.381221    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:40.704149 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.704109    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"]
Nov 01 10:07:40.704786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.704740    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" containerName="revoke-keys"
Nov 01 10:07:40.704786 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.704761    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" containerName="revoke-keys"
Nov 01 10:07:40.704930 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.704860    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" containerName="revoke-keys"
Nov 01 10:07:40.708178 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.708154    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.757753 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.757723    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:07:40.757753 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.757736    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:07:40.757983 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.757780    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:07:40.803837 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.803806    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:07:40.806254 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.806232    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"]
Nov 01 10:07:40.806580 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.806561    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.806633 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.806590    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.806706 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.806694    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgr6\" (UniqueName: \"kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.907891 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.907857    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgr6\" (UniqueName: \"kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.908062 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.907957    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.908062 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.907977    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.908673 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.908648    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.910381 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.910360    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:40.965837 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:40.965776    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgr6\" (UniqueName: \"kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6\") pod \"maas-api-revoke-sub-4fa569d2-llgs6\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:41.020704 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:41.020673    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:41.322833 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:07:41.322792    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:07:41.386202 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:41.386179    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"]
Nov 01 10:07:41.388677 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:07:41.388647    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c3dfc9_4537_4cfa_9603_9f09d15bca26.slice/crio-b5e9173bc5db5c352924f8821adfb519402e67f1a45f145f829e7ff4bd740f87 WatchSource:0}: Error finding container b5e9173bc5db5c352924f8821adfb519402e67f1a45f145f829e7ff4bd740f87: Status 404 returned error can't find the container with id b5e9173bc5db5c352924f8821adfb519402e67f1a45f145f829e7ff4bd740f87
Nov 01 10:07:41.887513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:41.887397    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6" event={"ID":"d0c3dfc9-4537-4cfa-9603-9f09d15bca26","Type":"ContainerStarted","Data":"4b64f050d7e09cc8fa4379256c401b615f2d1867f599a30730210127054cd79d"}
Nov 01 10:07:41.887513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:41.887465    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6" event={"ID":"d0c3dfc9-4537-4cfa-9603-9f09d15bca26","Type":"ContainerStarted","Data":"b5e9173bc5db5c352924f8821adfb519402e67f1a45f145f829e7ff4bd740f87"}
Nov 01 10:07:41.999083 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:41.999012    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6" podStartSLOduration=1.9989929850000001 podStartE2EDuration="1.998992985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:07:41.99813706 +0000 UTC m=+5735.192531616" watchObservedRunningTime="2026-11-01 10:07:41.998992985 +0000 UTC m=+5735.193387541"
Nov 01 10:07:42.897896 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:42.897854    4314 generic.go:356] "Generic (PLEG): container finished" podID="d0c3dfc9-4537-4cfa-9603-9f09d15bca26" containerID="4b64f050d7e09cc8fa4379256c401b615f2d1867f599a30730210127054cd79d" exitCode=0
Nov 01 10:07:42.898410 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:42.897913    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6" event={"ID":"d0c3dfc9-4537-4cfa-9603-9f09d15bca26","Type":"ContainerDied","Data":"4b64f050d7e09cc8fa4379256c401b615f2d1867f599a30730210127054cd79d"}
Nov 01 10:07:44.826556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.822522    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:44.840584 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.840555    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca\") pod \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") "
Nov 01 10:07:44.849089 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.848950    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca" pod "d0c3dfc9-4537-4cfa-9603-9f09d15bca26" (UID: "d0c3dfc9-4537-4cfa-9603-9f09d15bca26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:07:44.849722 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.849703    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca\") pod \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") "
Nov 01 10:07:44.851190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.849982    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6\" (UniqueName: \"kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6\") pod \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\" (UID: \"d0c3dfc9-4537-4cfa-9603-9f09d15bca26\") "
Nov 01 10:07:44.851808 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.851787    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:44.865018 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.864980    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6" pod "d0c3dfc9-4537-4cfa-9603-9f09d15bca26" (UID: "d0c3dfc9-4537-4cfa-9603-9f09d15bca26"). InnerVolumeSpecName "kube-api-access-4mgr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:07:44.907573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.905640    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca" pod "d0c3dfc9-4537-4cfa-9603-9f09d15bca26" (UID: "d0c3dfc9-4537-4cfa-9603-9f09d15bca26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:07:44.977335 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.977301    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:44.998685 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:44.998595    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mgr6\" (UniqueName: \"kubernetes.io/projected/d0c3dfc9-4537-4cfa-9603-9f09d15bca26-kube-api-access-4mgr6\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:45.041762 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:45.041725    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6" event={"ID":"d0c3dfc9-4537-4cfa-9603-9f09d15bca26","Type":"ContainerDied","Data":"b5e9173bc5db5c352924f8821adfb519402e67f1a45f145f829e7ff4bd740f87"}
Nov 01 10:07:45.042018 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:45.041997    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e9173bc5db5c352924f8821adfb519402e67f1a45f145f829e7ff4bd740f87"
Nov 01 10:07:45.048106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:45.041762    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"
Nov 01 10:07:50.226356 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.226322    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc" podUID="9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" containerName="main" containerID="cri-o://3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4" gracePeriod=45
Nov 01 10:07:50.471746 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.471720    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:07:50.512571 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512525    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir\") pod \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") "
Nov 01 10:07:50.512708 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512667    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs\") pod \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") "
Nov 01 10:07:50.512784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512707    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm\") pod \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") "
Nov 01 10:07:50.512784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512747    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir" pod "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" (UID: "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:07:50.512784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512757    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home\") pod \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") "
Nov 01 10:07:50.512958 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512802    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw\" (UniqueName: \"kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw\") pod \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") "
Nov 01 10:07:50.512958 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512847    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache\") pod \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\" (UID: \"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06\") "
Nov 01 10:07:50.513078 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.512968    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home" pod "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" (UID: "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:07:50.513183 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.513161    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache" pod "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" (UID: "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:07:50.513279 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.513262    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:50.513323 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.513286    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:50.513379 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.513326    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:50.515006 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.514971    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm" pod "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" (UID: "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:07:50.515087 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.515043    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs" pod "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" (UID: "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:07:50.515130 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.515081    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw" pod "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" (UID: "9fbca358-6ef0-42a2-9f39-5f5a11a9ca06"). InnerVolumeSpecName "kube-api-access-qr9gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:07:50.614378 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.614349    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr9gw\" (UniqueName: \"kubernetes.io/projected/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-kube-api-access-qr9gw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:50.614378 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.614375    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:50.614378 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:50.614384    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:07:51.074126 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.074095    4314 generic.go:356] "Generic (PLEG): container finished" podID="9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" containerID="3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4" exitCode=0
Nov 01 10:07:51.074333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.074146    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"
Nov 01 10:07:51.074333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.074171    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc" event={"ID":"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06","Type":"ContainerDied","Data":"3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4"}
Nov 01 10:07:51.074333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.074217    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc" event={"ID":"9fbca358-6ef0-42a2-9f39-5f5a11a9ca06","Type":"ContainerDied","Data":"43372096819f591a16fab7652bba68c97db12de978cc397b78b25bff61e9cb55"}
Nov 01 10:07:51.074333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.074234    4314 scope.go:122] "RemoveContainer" containerID="3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4"
Nov 01 10:07:51.083378 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.083350    4314 scope.go:122] "RemoveContainer" containerID="3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4"
Nov 01 10:07:51.083676 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:07:51.083648    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4\": container with ID starting with 3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4 not found: ID does not exist" containerID="3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4"
Nov 01 10:07:51.083784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.083677    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4"} err="failed to get container status \"3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4\": rpc error: code = NotFound desc = could not find container \"3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4\": container with ID starting with 3a33fdfdc3bf0b95e07dd5a2541056178bc3682d0a3394f684d591ce01995cc4 not found: ID does not exist"
Nov 01 10:07:51.227185 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.227122    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"]
Nov 01 10:07:51.273584 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:51.273554    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-65f8cfb469-br5wc"]
Nov 01 10:07:52.927630 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:52.927600    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" path="/var/lib/kubelet/pods/9fbca358-6ef0-42a2-9f39-5f5a11a9ca06/volumes"
Nov 01 10:07:53.312366 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.312334    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:53.748881 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.748848    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"]
Nov 01 10:07:53.749234 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.749221    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d0c3dfc9-4537-4cfa-9603-9f09d15bca26" containerName="revoke-keys"
Nov 01 10:07:53.749281 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.749236    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c3dfc9-4537-4cfa-9603-9f09d15bca26" containerName="revoke-keys"
Nov 01 10:07:53.749281 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.749253    4314 cpu_manager.go:414] "containerMap: removing container" podUID="9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" containerName="main"
Nov 01 10:07:53.749281 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.749258    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" containerName="main"
Nov 01 10:07:53.749381 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.749332    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbca358-6ef0-42a2-9f39-5f5a11a9ca06" containerName="main"
Nov 01 10:07:53.749381 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.749346    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c3dfc9-4537-4cfa-9603-9f09d15bca26" containerName="revoke-keys"
Nov 01 10:07:53.753709 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.753689    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.803559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.803496    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-029e39-openshift-default-dockercfg-mlzdt\""
Nov 01 10:07:53.843270 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843236    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843270 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843270    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843298    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843432    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843488    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843660 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843571    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkzf\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843660 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843621    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843782 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843697    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.843782 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.843768    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.855660 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.855623    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"]
Nov 01 10:07:53.858428 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.858403    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:53.944344 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944315    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944366    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkzf\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944448    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944489    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944554    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944631    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944660    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944690    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944725    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.944807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944732    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.945690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.944862    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.945690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.945068    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.945838 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.945818    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.946066 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.946046    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.947516 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.947490    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:53.947691 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:53.947669    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:54.001911 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.001837    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:54.002052 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.001956    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkzf\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf\") pod \"e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") " pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:54.010868 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.010844    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:54.065795 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.065767    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:54.483409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.483384    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"]
Nov 01 10:07:54.486475 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:07:54.486443    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b817b2_df0a_420c_8754_713074e09809.slice/crio-4a15bc1636efeb59794ad049cf0a8a426e1f076e97d83e6faa5e319c622b3ee1 WatchSource:0}: Error finding container 4a15bc1636efeb59794ad049cf0a8a426e1f076e97d83e6faa5e319c622b3ee1: Status 404 returned error can't find the container with id 4a15bc1636efeb59794ad049cf0a8a426e1f076e97d83e6faa5e319c622b3ee1
Nov 01 10:07:54.488989 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.488954    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:07:54.489109 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.489027    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:07:54.489109 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:54.489062    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:07:55.111614 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:55.109398    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" event={"ID":"33b817b2-df0a-420c-8754-713074e09809","Type":"ContainerStarted","Data":"7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"}
Nov 01 10:07:55.111614 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:55.109469    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" event={"ID":"33b817b2-df0a-420c-8754-713074e09809","Type":"ContainerStarted","Data":"4a15bc1636efeb59794ad049cf0a8a426e1f076e97d83e6faa5e319c622b3ee1"}
Nov 01 10:07:56.080361 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:56.080324    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:56.088815 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:56.088789    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:56.125450 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:56.125289    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:56.128814 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:56.128748    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:07:56.224333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:56.224254    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podStartSLOduration=3.224224537 podStartE2EDuration="3.224224537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:07:55.258934799 +0000 UTC m=+5748.453329356" watchObservedRunningTime="2026-11-01 10:07:56.224224537 +0000 UTC m=+5749.418619093"
Nov 01 10:07:58.225340 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:58.225302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:58.389492 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:58.389461    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:07:58.546871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:07:58.546812    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:01.430689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.430655    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:01.609370 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.609337    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:01.853865 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.853829    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"]
Nov 01 10:08:01.859286 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.859261    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:01.910266 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.910238    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"]
Nov 01 10:08:01.953872 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.953840    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-029e39\""
Nov 01 10:08:01.954095 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.953954    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ccf3ac19-dockercfg-n9zp7\""
Nov 01 10:08:01.977040 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.977007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsx6j\" (UniqueName: \"kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j\") pod \"payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:01.977211 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:01.977146    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:02.002513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.002485    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"]
Nov 01 10:08:02.006034 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.006019    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.037004 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.036980    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"]
Nov 01 10:08:02.077682 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.077658    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.077812 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.077702    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:02.077862 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.077823    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsx6j\" (UniqueName: \"kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j\") pod \"payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:02.077862 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.077858    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d28k\" (UniqueName: \"kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k\") pod \"payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.078439 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.078421    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:02.135949 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.135890    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsx6j\" (UniqueName: \"kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j\") pod \"payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:02.172241 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.172222    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:02.178969 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.178947    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d28k\" (UniqueName: \"kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k\") pod \"payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.179050 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.179033    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.179722 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.179702    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.240557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.238740    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d28k\" (UniqueName: \"kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k\") pod \"payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.318427 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.318397    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:02.537611 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.537586    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"]
Nov 01 10:08:02.539506 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:08:02.539476    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2adeeb_7856_4c58_9f07_4ea21318d25d.slice/crio-6e7774eca969c4a4a911b70ec8735c72b8573100de2b5d215104cf785e696ffb WatchSource:0}: Error finding container 6e7774eca969c4a4a911b70ec8735c72b8573100de2b5d215104cf785e696ffb: Status 404 returned error can't find the container with id 6e7774eca969c4a4a911b70ec8735c72b8573100de2b5d215104cf785e696ffb
Nov 01 10:08:02.686043 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:02.686016    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"]
Nov 01 10:08:02.687482 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:08:02.687460    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73bf34f_81cf_47f7_a355_7f1140c2f6e5.slice/crio-51ed688db579063bd8b56b48f4357efa5b9405eaf8494f4efc019b27beeca7e3 WatchSource:0}: Error finding container 51ed688db579063bd8b56b48f4357efa5b9405eaf8494f4efc019b27beeca7e3: Status 404 returned error can't find the container with id 51ed688db579063bd8b56b48f4357efa5b9405eaf8494f4efc019b27beeca7e3
Nov 01 10:08:03.170055 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.170021    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" event={"ID":"d73bf34f-81cf-47f7-a355-7f1140c2f6e5","Type":"ContainerStarted","Data":"8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925"}
Nov 01 10:08:03.170250 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.170075    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" event={"ID":"d73bf34f-81cf-47f7-a355-7f1140c2f6e5","Type":"ContainerStarted","Data":"51ed688db579063bd8b56b48f4357efa5b9405eaf8494f4efc019b27beeca7e3"}
Nov 01 10:08:03.170250 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.170180    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:03.171569 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.171524    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" event={"ID":"dd2adeeb-7856-4c58-9f07-4ea21318d25d","Type":"ContainerStarted","Data":"69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c"}
Nov 01 10:08:03.171961 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.171588    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" event={"ID":"dd2adeeb-7856-4c58-9f07-4ea21318d25d","Type":"ContainerStarted","Data":"6e7774eca969c4a4a911b70ec8735c72b8573100de2b5d215104cf785e696ffb"}
Nov 01 10:08:03.171961 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.171634    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:03.289282 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.289215    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" podStartSLOduration=2.288981225 podStartE2EDuration="2.288981225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:08:03.288811261 +0000 UTC m=+5756.483205816" watchObservedRunningTime="2026-11-01 10:08:03.288981225 +0000 UTC m=+5756.483375774"
Nov 01 10:08:03.400109 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:03.400056    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" podStartSLOduration=2.400042764 podStartE2EDuration="2.400042764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:08:03.398728526 +0000 UTC m=+5756.593123080" watchObservedRunningTime="2026-11-01 10:08:03.400042764 +0000 UTC m=+5756.594437319"
Nov 01 10:08:04.124214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:04.124178    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:04.275203 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:04.275163    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:04.425857 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:04.425766    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:08.865921 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:08.865884    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:11.322084 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:08:11.322051    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:08:11.581339 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:11.581272    4314 scope.go:122] "RemoveContainer" containerID="5a2bb06a39264b573612ad74f09caee9e67d106d46016e20d9f54b45af7479a0"
Nov 01 10:08:11.589789 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:11.589769    4314 scope.go:122] "RemoveContainer" containerID="584ab4de6cab4a662e691a800769e5264b086f1b87823d9e100af60ade8d371c"
Nov 01 10:08:13.488363 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:13.488299    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:14.183347 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:14.183306    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:08:14.183629 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:14.183603    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:08:18.079876 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:18.079803    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:21.274012 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:21.273982    4314 generic.go:356] "Generic (PLEG): container finished" podID="33b817b2-df0a-420c-8754-713074e09809" containerID="7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf" exitCode=0
Nov 01 10:08:21.274411 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:21.274058    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" event={"ID":"33b817b2-df0a-420c-8754-713074e09809","Type":"ContainerDied","Data":"7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"}
Nov 01 10:08:21.274461 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:21.274448    4314 scope.go:122] "RemoveContainer" containerID="7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"
Nov 01 10:08:21.275595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:21.275565    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:08:21.275743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:21.275631    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:08:21.275743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:21.275662    4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 10:08:22.290933 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:22.290898    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" event={"ID":"33b817b2-df0a-420c-8754-713074e09809","Type":"ContainerStarted","Data":"ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a"}
Nov 01 10:08:22.909549 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:22.909504    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:23.071886 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:23.071852    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:08:23.082687 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:23.082645    4314 patch_prober.go:29] interesting pod/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body=
Nov 01 10:08:23.082864 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:23.082722    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused"
Nov 01 10:08:23.083940 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:23.083915    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:23.719462 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:23.719427    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:23.881010 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:23.880979    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:24.074358 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:24.073916    4314 patch_prober.go:29] interesting pod/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body=
Nov 01 10:08:24.074358 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:24.073979    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused"
Nov 01 10:08:25.077557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:25.073602    4314 patch_prober.go:29] interesting pod/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body=
Nov 01 10:08:25.077557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:25.073680    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused"
Nov 01 10:08:26.072972 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:26.072932    4314 patch_prober.go:29] interesting pod/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body=
Nov 01 10:08:26.073305 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:26.073271    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused"
Nov 01 10:08:26.125756 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:26.125727    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:08:26.828440 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:26.828408    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:27.070714 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:27.070676    4314 patch_prober.go:29] interesting pod/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body=
Nov 01 10:08:27.070996 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:27.070965    4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused"
Nov 01 10:08:27.692528 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:27.692473    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:28.070695 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:28.070661    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:08:28.071705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:28.071684    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:08:32.380045 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:32.380013    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:33.068934 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:33.068897    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:36.948416 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:36.948382    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:41.322148 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:08:41.322106    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:08:41.553941 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:41.553905    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:46.051277 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:46.051244    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:50.652456 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:50.652378    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:55.170834 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:55.170801    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:08:59.758051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:08:59.758011    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:04.342803 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:04.342767    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:08.971620 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:08.971589    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:11.321480 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:09:11.321446    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:09:13.024277 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:13.024244    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:13.485685 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:13.485647    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:18.053938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:18.053898    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:22.634364 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:22.634333    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:24.046801 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:24.046764    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"]
Nov 01 10:09:24.092622 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:24.092590    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7hdcb"]
Nov 01 10:09:24.211457 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:24.211424    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:24.661165 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:24.661134    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:24.927682 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:24.927610    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3375511-49ce-49ce-b37b-b93e5b823c87" path="/var/lib/kubelet/pods/a3375511-49ce-49ce-b37b-b93e5b823c87/volumes"
Nov 01 10:09:27.237028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:27.236993    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:29.259594 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:29.259554    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:30.628740 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:30.628705    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:31.423004 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:31.422962    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"]
Nov 01 10:09:31.532798 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:31.532759    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"]
Nov 01 10:09:31.794893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:31.794846    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:31.945126 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:31.945092    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:33.407735 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:33.407687    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:36.128523 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:36.128496    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:36.423953 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:36.423847    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerName="payload-processing" containerID="cri-o://8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925" gracePeriod=25
Nov 01 10:09:36.533549 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:36.533491    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerName="payload-pre-processing" containerID="cri-o://69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c" gracePeriod=25
Nov 01 10:09:36.565936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:36.565907    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:41.322139 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:09:41.322097    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:09:42.501820 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:42.501791    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:42.659352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:42.659315    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:44.177905 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:44.177864    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:09:44.178311 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:44.177864    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:09:51.119043 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:51.119010    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:51.260175 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:51.260139    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:09:54.178062 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:54.178028    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:09:54.178462 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:54.178069    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:09:56.679349 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.679293    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:09:56.752661 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.752632    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k\" (UniqueName: \"kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k\") pod \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") "
Nov 01 10:09:56.752835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.752747    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume\") pod \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\" (UID: \"d73bf34f-81cf-47f7-a355-7f1140c2f6e5\") "
Nov 01 10:09:56.753077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.753049    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume" pod "d73bf34f-81cf-47f7-a355-7f1140c2f6e5" (UID: "d73bf34f-81cf-47f7-a355-7f1140c2f6e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:09:56.753224 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.753160    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:56.754806 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.754783    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k" pod "d73bf34f-81cf-47f7-a355-7f1140c2f6e5" (UID: "d73bf34f-81cf-47f7-a355-7f1140c2f6e5"). InnerVolumeSpecName "kube-api-access-2d28k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:09:56.769647 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.769628    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:09:56.777515 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.777488    4314 generic.go:356] "Generic (PLEG): container finished" podID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerID="8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925" exitCode=0
Nov 01 10:09:56.777639 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.777553    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"
Nov 01 10:09:56.777639 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.777562    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" event={"ID":"d73bf34f-81cf-47f7-a355-7f1140c2f6e5","Type":"ContainerDied","Data":"8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925"}
Nov 01 10:09:56.777639 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.777615    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft" event={"ID":"d73bf34f-81cf-47f7-a355-7f1140c2f6e5","Type":"ContainerDied","Data":"51ed688db579063bd8b56b48f4357efa5b9405eaf8494f4efc019b27beeca7e3"}
Nov 01 10:09:56.777815 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.777645    4314 scope.go:122] "RemoveContainer" containerID="8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925"
Nov 01 10:09:56.779028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.778945    4314 generic.go:356] "Generic (PLEG): container finished" podID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerID="69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c" exitCode=0
Nov 01 10:09:56.779028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.778973    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" event={"ID":"dd2adeeb-7856-4c58-9f07-4ea21318d25d","Type":"ContainerDied","Data":"69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c"}
Nov 01 10:09:56.779028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.779015    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s" event={"ID":"dd2adeeb-7856-4c58-9f07-4ea21318d25d","Type":"ContainerDied","Data":"6e7774eca969c4a4a911b70ec8735c72b8573100de2b5d215104cf785e696ffb"}
Nov 01 10:09:56.779028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.778983    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"
Nov 01 10:09:56.786243 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.786228    4314 scope.go:122] "RemoveContainer" containerID="8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925"
Nov 01 10:09:56.786506 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:09:56.786486    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925\": container with ID starting with 8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925 not found: ID does not exist" containerID="8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925"
Nov 01 10:09:56.786621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.786517    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925"} err="failed to get container status \"8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925\": rpc error: code = NotFound desc = could not find container \"8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925\": container with ID starting with 8d764dda920dd65d39da99c8187b2c1f76289ff7baefd1b7b4a8e339a5913925 not found: ID does not exist"
Nov 01 10:09:56.786621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.786562    4314 scope.go:122] "RemoveContainer" containerID="69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c"
Nov 01 10:09:56.794018 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.794003    4314 scope.go:122] "RemoveContainer" containerID="69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c"
Nov 01 10:09:56.794264 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:09:56.794248    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c\": container with ID starting with 69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c not found: ID does not exist" containerID="69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c"
Nov 01 10:09:56.794323 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.794269    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c"} err="failed to get container status \"69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c\": rpc error: code = NotFound desc = could not find container \"69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c\": container with ID starting with 69238ba3f3daf48aba2d5f798a1a45dd98c39434f779ea82d8891b1884cbe75c not found: ID does not exist"
Nov 01 10:09:56.853672 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.853650    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j\" (UniqueName: \"kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j\") pod \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") "
Nov 01 10:09:56.853780 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.853738    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume\") pod \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\" (UID: \"dd2adeeb-7856-4c58-9f07-4ea21318d25d\") "
Nov 01 10:09:56.853995 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.853979    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d28k\" (UniqueName: \"kubernetes.io/projected/d73bf34f-81cf-47f7-a355-7f1140c2f6e5-kube-api-access-2d28k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:56.854115 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.854056    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume" pod "dd2adeeb-7856-4c58-9f07-4ea21318d25d" (UID: "dd2adeeb-7856-4c58-9f07-4ea21318d25d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:09:56.855457 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.855440    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j" pod "dd2adeeb-7856-4c58-9f07-4ea21318d25d" (UID: "dd2adeeb-7856-4c58-9f07-4ea21318d25d"). InnerVolumeSpecName "kube-api-access-nsx6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:09:56.955272 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.955242    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsx6j\" (UniqueName: \"kubernetes.io/projected/dd2adeeb-7856-4c58-9f07-4ea21318d25d-kube-api-access-nsx6j\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:56.955272 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:56.955266    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2adeeb-7856-4c58-9f07-4ea21318d25d-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:57.021255 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:57.021218    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"]
Nov 01 10:09:57.026306 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:57.026279    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-029e39-d4cdfff67sf2ft"]
Nov 01 10:09:57.186243 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:57.186176    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"]
Nov 01 10:09:57.190505 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:57.190480    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-029e39-6c4c8dkh8s"]
Nov 01 10:09:58.927419 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:58.927377    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" path="/var/lib/kubelet/pods/d73bf34f-81cf-47f7-a355-7f1140c2f6e5/volumes"
Nov 01 10:09:58.927969 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:09:58.927928    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" path="/var/lib/kubelet/pods/dd2adeeb-7856-4c58-9f07-4ea21318d25d/volumes"
Nov 01 10:10:06.335089 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:06.335052    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:06.504144 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:06.504113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:07.922145 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:07.922116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:08.525844 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:08.525813    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"]
Nov 01 10:10:08.526084 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:08.526038    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" containerID="cri-o://ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a" gracePeriod=30
Nov 01 10:10:08.541294 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:08.541264    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:11.321968 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:10:11.321931    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:10:11.623893 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:11.623818    4314 scope.go:122] "RemoveContainer" containerID="68d364e141f9d819fd53fa1de79225abbee721420698c4b2518c7eb418451b2b"
Nov 01 10:10:12.661205 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:12.661168    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:12.829189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:12.829129    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:13.073120 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.073087    4314 patch_prober.go:29] interesting pod/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 10:10:13.073292 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.073148    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 10:10:13.777045 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.777017    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:10:13.813135 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813075    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813135 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813107    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813135 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813136    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813173    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813243    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813275    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813303    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813593 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813422    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813593 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813471    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token\") pod \"33b817b2-df0a-420c-8754-713074e09809\" (UID: \"33b817b2-df0a-420c-8754-713074e09809\") "
Nov 01 10:10:13.813593 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813504    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:10:13.813783 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813726    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:10:13.813783 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813748    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:10:13.813896 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813834    4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-credential-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.813896 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.813856    4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-data\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.814103 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.814068    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:10:13.814233 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.814198    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:10:13.815454 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.815425    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:10:13.815768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.815748    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:10:13.815989 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.815967    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "kube-api-access-fzkzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:10:13.815989 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.815972    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy" pod "33b817b2-df0a-420c-8754-713074e09809" (UID: "33b817b2-df0a-420c-8754-713074e09809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:10:13.865052 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.865023    4314 generic.go:356] "Generic (PLEG): container finished" podID="33b817b2-df0a-420c-8754-713074e09809" containerID="ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a" exitCode=0
Nov 01 10:10:13.865214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.865073    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"
Nov 01 10:10:13.865214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.865115    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" event={"ID":"33b817b2-df0a-420c-8754-713074e09809","Type":"ContainerDied","Data":"ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a"}
Nov 01 10:10:13.865214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.865172    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d" event={"ID":"33b817b2-df0a-420c-8754-713074e09809","Type":"ContainerDied","Data":"4a15bc1636efeb59794ad049cf0a8a426e1f076e97d83e6faa5e319c622b3ee1"}
Nov 01 10:10:13.865214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.865196    4314 scope.go:122] "RemoveContainer" containerID="ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a"
Nov 01 10:10:13.874792 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.874648    4314 scope.go:122] "RemoveContainer" containerID="7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"
Nov 01 10:10:13.882437 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.882420    4314 scope.go:122] "RemoveContainer" containerID="ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a"
Nov 01 10:10:13.882709 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:10:13.882693    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a\": container with ID starting with ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a not found: ID does not exist" containerID="ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a"
Nov 01 10:10:13.882772 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.882717    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a"} err="failed to get container status \"ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a\": rpc error: code = NotFound desc = could not find container \"ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a\": container with ID starting with ae13b216c66928be2d2ca5018c942e7bf78be6d0aaf0636be952c20ebee8595a not found: ID does not exist"
Nov 01 10:10:13.882772 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.882731    4314 scope.go:122] "RemoveContainer" containerID="7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"
Nov 01 10:10:13.882978 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:10:13.882962    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf\": container with ID starting with 7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf not found: ID does not exist" containerID="7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"
Nov 01 10:10:13.883021 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.882984    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf"} err="failed to get container status \"7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf\": rpc error: code = NotFound desc = could not find container \"7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf\": container with ID starting with 7ba0c667f4a03e0db3085e1bab6ffb7af5a00b8756428824979d7bc1065977cf not found: ID does not exist"
Nov 01 10:10:13.914677 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914656    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzkzf\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-kube-api-access-fzkzf\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.914773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914678    4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.914773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914689    4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-istio-envoy\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.914773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914698    4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b817b2-df0a-420c-8754-713074e09809-istio-token\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.914773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914707    4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b817b2-df0a-420c-8754-713074e09809-istio-podinfo\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.914773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914715    4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b817b2-df0a-420c-8754-713074e09809-workload-socket\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:13.914773 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:13.914723    4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b817b2-df0a-420c-8754-713074e09809-istiod-ca-cert\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:10:14.041652 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:14.041622    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"]
Nov 01 10:10:14.048433 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:14.048409    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-029e39-openshift-default-5c68cd4b8fx698d"]
Nov 01 10:10:14.926489 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:14.926458    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b817b2-df0a-420c-8754-713074e09809" path="/var/lib/kubelet/pods/33b817b2-df0a-420c-8754-713074e09809/volumes"
Nov 01 10:10:26.510268 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510188    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"]
Nov 01 10:10:26.510705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510614    4314 cpu_manager.go:414] "containerMap: removing container" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy"
Nov 01 10:10:26.510705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510626    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy"
Nov 01 10:10:26.510705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510650    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerName="payload-processing"
Nov 01 10:10:26.510705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510655    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerName="payload-processing"
Nov 01 10:10:26.510705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510665    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerName="payload-pre-processing"
Nov 01 10:10:26.510705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510670    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerName="payload-pre-processing"
Nov 01 10:10:26.510941 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510736    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy"
Nov 01 10:10:26.510941 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510747    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73bf34f-81cf-47f7-a355-7f1140c2f6e5" containerName="payload-processing"
Nov 01 10:10:26.510941 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510755    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2adeeb-7856-4c58-9f07-4ea21318d25d" containerName="payload-pre-processing"
Nov 01 10:10:26.510941 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.510764    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy"
Nov 01 10:10:26.515460 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.515443    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.611086 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.611051    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:26.612170 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.612153    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-46760c40-kserve-self-signed-certs\""
Nov 01 10:10:26.620551 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.615252    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"]
Nov 01 10:10:26.636164 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.636126    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.636317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.636171    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndps7\" (UniqueName: \"kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.636317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.636257    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.636408 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.636311    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.636447 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.636401    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.636447 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.636439    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.737986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.737941    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.737986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.737991    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndps7\" (UniqueName: \"kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738020    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738046    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738087    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738129    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738515 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738444    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738515 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738479    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.738633 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.738528    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.740353 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.740330    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.740730 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.740712    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.795655 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.795574    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndps7\" (UniqueName: \"kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7\") pod \"e2e-same-id-a-46760c40-kserve-56687867fc-cd29z\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") " pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:26.827387 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:26.827353    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:27.200928 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:27.200733    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"]
Nov 01 10:10:27.203145 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:10:27.203118    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd510f63d_5e04_41af_a186_0b3014af2ddf.slice/crio-416b8959b93a8fb59ca4e27d012121e58a184f22b3de151a56cccbc4c196c8e9 WatchSource:0}: Error finding container 416b8959b93a8fb59ca4e27d012121e58a184f22b3de151a56cccbc4c196c8e9: Status 404 returned error can't find the container with id 416b8959b93a8fb59ca4e27d012121e58a184f22b3de151a56cccbc4c196c8e9
Nov 01 10:10:27.943639 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:27.943607    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" event={"ID":"d510f63d-5e04-41af-a186-0b3014af2ddf","Type":"ContainerStarted","Data":"f6e4fa63e9f401be77a0ec772797d222daae3fcec39643475ba4fd64b3031f21"}
Nov 01 10:10:27.944046 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:27.943647    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" event={"ID":"d510f63d-5e04-41af-a186-0b3014af2ddf","Type":"ContainerStarted","Data":"416b8959b93a8fb59ca4e27d012121e58a184f22b3de151a56cccbc4c196c8e9"}
Nov 01 10:10:28.054944 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:28.054891    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" podStartSLOduration=2.054876949 podStartE2EDuration="2.054876949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:10:28.053597856 +0000 UTC m=+5901.247992413" watchObservedRunningTime="2026-11-01 10:10:28.054876949 +0000 UTC m=+5901.249271503"
Nov 01 10:10:28.828120 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:28.828082    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:33.791407 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:33.791375    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:10:34.804557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:34.803622    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:10:34.804557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:34.803689    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:10:36.828317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:36.828280    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:36.840875 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:36.840849    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:37.050401 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:37.050370    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:10:41.322104 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:10:41.322068    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:10:53.849811 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:10:53.849775    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:11:02.183613 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:02.180643    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:11:02.183613 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:02.180738    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:11:02.451814 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:02.451716    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:11:02.452072 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:02.451818    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:11:11.322609 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:11:11.322568    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:11:11.655427 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:11.655345    4314 scope.go:122] "RemoveContainer" containerID="a5002344982e9756207c1ce859e3785ee281512efabbe5ad9384e39be2730b46"
Nov 01 10:11:27.365547 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.365507    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"]
Nov 01 10:11:27.366097 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.365900    4314 cpu_manager.go:414] "containerMap: removing container" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy"
Nov 01 10:11:27.366097 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.365911    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b817b2-df0a-420c-8754-713074e09809" containerName="istio-proxy"
Nov 01 10:11:27.370631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.370610    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.423860 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.423828    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"]
Nov 01 10:11:27.453269 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.453236    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.453491 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.453283    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndpr\" (UniqueName: \"kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.453491 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.453458    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.466900 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.466872    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:11:27.467084 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.466888    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:11:27.467084 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.466895    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:11:27.467084 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.466930    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:11:27.554674 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.554629    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.554674 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.554686    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndpr\" (UniqueName: \"kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.554987 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.554734    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.555424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.555399    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.557147 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.557121    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.613433 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.613402    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndpr\" (UniqueName: \"kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr\") pod \"maas-api-revoke-sub-452b6d7b-zxfw2\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:27.683712 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:27.683630    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:28.052566 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:28.052514    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"]
Nov 01 10:11:28.054711 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:11:28.054678    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3d8a18_72b4_4158_8869_a6dc0141e657.slice/crio-751cdce144e3d3d684464196121fba22c48939f6ea9bd634d8a00b16c48057be WatchSource:0}: Error finding container 751cdce144e3d3d684464196121fba22c48939f6ea9bd634d8a00b16c48057be: Status 404 returned error can't find the container with id 751cdce144e3d3d684464196121fba22c48939f6ea9bd634d8a00b16c48057be
Nov 01 10:11:28.323322 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:28.323238    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2" event={"ID":"7e3d8a18-72b4-4158-8869-a6dc0141e657","Type":"ContainerStarted","Data":"751cdce144e3d3d684464196121fba22c48939f6ea9bd634d8a00b16c48057be"}
Nov 01 10:11:28.565528 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:28.565492    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:11:28.754458 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:28.754421    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:11:29.333392 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:29.333352    4314 generic.go:356] "Generic (PLEG): container finished" podID="7e3d8a18-72b4-4158-8869-a6dc0141e657" containerID="d73a5ded0c142d81347e7732212b3b48312178ecce5b6a643d04fcc8d790bb40" exitCode=0
Nov 01 10:11:29.333614 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:29.333458    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2" event={"ID":"7e3d8a18-72b4-4158-8869-a6dc0141e657","Type":"ContainerDied","Data":"d73a5ded0c142d81347e7732212b3b48312178ecce5b6a643d04fcc8d790bb40"}
Nov 01 10:11:30.745575 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.741863    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:30.836557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.835897    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca\") pod \"7e3d8a18-72b4-4158-8869-a6dc0141e657\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") "
Nov 01 10:11:30.836557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.836019    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca\") pod \"7e3d8a18-72b4-4158-8869-a6dc0141e657\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") "
Nov 01 10:11:30.836557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.836121    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr\" (UniqueName: \"kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr\") pod \"7e3d8a18-72b4-4158-8869-a6dc0141e657\" (UID: \"7e3d8a18-72b4-4158-8869-a6dc0141e657\") "
Nov 01 10:11:30.840586 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.837031    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca" pod "7e3d8a18-72b4-4158-8869-a6dc0141e657" (UID: "7e3d8a18-72b4-4158-8869-a6dc0141e657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:11:30.844556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.842588    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca" pod "7e3d8a18-72b4-4158-8869-a6dc0141e657" (UID: "7e3d8a18-72b4-4158-8869-a6dc0141e657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:11:30.848555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.844971    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr" pod "7e3d8a18-72b4-4158-8869-a6dc0141e657" (UID: "7e3d8a18-72b4-4158-8869-a6dc0141e657"). InnerVolumeSpecName "kube-api-access-jndpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:11:30.953120 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.951742    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndpr\" (UniqueName: \"kubernetes.io/projected/7e3d8a18-72b4-4158-8869-a6dc0141e657-kube-api-access-jndpr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:30.953120 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.951784    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:30.953120 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:30.951799    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e3d8a18-72b4-4158-8869-a6dc0141e657-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:31.382569 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:31.381353    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2" event={"ID":"7e3d8a18-72b4-4158-8869-a6dc0141e657","Type":"ContainerDied","Data":"751cdce144e3d3d684464196121fba22c48939f6ea9bd634d8a00b16c48057be"}
Nov 01 10:11:31.382569 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:31.381457    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="751cdce144e3d3d684464196121fba22c48939f6ea9bd634d8a00b16c48057be"
Nov 01 10:11:31.382569 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:31.381573    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"
Nov 01 10:11:32.286379 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:32.286320    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:11:32.287079 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:32.287038    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:11:32.457573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:32.457370    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:11:32.457573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:32.457454    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:11:34.181221 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:34.181182    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:11:40.204071 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:40.204034    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"]
Nov 01 10:11:40.216103 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:40.216072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:11:41.322613 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:11:41.322526    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:11:48.549696 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:48.549657    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:11:53.766242 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.766205    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"]
Nov 01 10:11:53.766673 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.766648    4314 cpu_manager.go:414] "containerMap: removing container" podUID="7e3d8a18-72b4-4158-8869-a6dc0141e657" containerName="revoke-keys"
Nov 01 10:11:53.766673 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.766662    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3d8a18-72b4-4158-8869-a6dc0141e657" containerName="revoke-keys"
Nov 01 10:11:53.766753 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.766745    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3d8a18-72b4-4158-8869-a6dc0141e657" containerName="revoke-keys"
Nov 01 10:11:53.770156 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.770140    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:53.821167 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.821133    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:11:53.824417 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.824382    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"]
Nov 01 10:11:53.866768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.866733    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:11:53.866768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.866767    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:11:53.867015 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.866855    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:11:53.912368 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.912311    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:53.912629 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.912522    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q2rc\" (UniqueName: \"kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:53.912629 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:53.912595    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.014689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.013588    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.014689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.013749    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.014689 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.013887    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q2rc\" (UniqueName: \"kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.015252 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.015221    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.017865 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.017190    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.073317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.073248    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q2rc\" (UniqueName: \"kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc\") pod \"maas-api-revoke-sub-3e31a506-2qk7c\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.090995 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.090959    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:54.483523 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.483470    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"]
Nov 01 10:11:54.488153 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:11:54.488113    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fb7541_2600_49b1_bab8_719215160d51.slice/crio-e61471cf01dfedf044c5ba944bf230ce0181fbb20b4ddb698a89dcc7b8580e29 WatchSource:0}: Error finding container e61471cf01dfedf044c5ba944bf230ce0181fbb20b4ddb698a89dcc7b8580e29: Status 404 returned error can't find the container with id e61471cf01dfedf044c5ba944bf230ce0181fbb20b4ddb698a89dcc7b8580e29
Nov 01 10:11:54.589173 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:54.589123    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c" event={"ID":"20fb7541-2600-49b1-bab8-719215160d51","Type":"ContainerStarted","Data":"e61471cf01dfedf044c5ba944bf230ce0181fbb20b4ddb698a89dcc7b8580e29"}
Nov 01 10:11:55.232626 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:55.231520    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" podUID="d510f63d-5e04-41af-a186-0b3014af2ddf" containerName="main" containerID="cri-o://f6e4fa63e9f401be77a0ec772797d222daae3fcec39643475ba4fd64b3031f21" gracePeriod=45
Nov 01 10:11:55.628860 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:55.627813    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c" event={"ID":"20fb7541-2600-49b1-bab8-719215160d51","Type":"ContainerStarted","Data":"186a59595ec53762a67f4d149bac4cce42bd77a94f107656eb3136613e7f53a5"}
Nov 01 10:11:55.658214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:55.658023    4314 generic.go:356] "Generic (PLEG): container finished" podID="d510f63d-5e04-41af-a186-0b3014af2ddf" containerID="f6e4fa63e9f401be77a0ec772797d222daae3fcec39643475ba4fd64b3031f21" exitCode=0
Nov 01 10:11:55.658214 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:55.658149    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" event={"ID":"d510f63d-5e04-41af-a186-0b3014af2ddf","Type":"ContainerDied","Data":"f6e4fa63e9f401be77a0ec772797d222daae3fcec39643475ba4fd64b3031f21"}
Nov 01 10:11:56.265560 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.262716    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:11:56.381154 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.374309    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c" podStartSLOduration=3.374286208 podStartE2EDuration="3.374286208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:11:55.751163731 +0000 UTC m=+5988.945558287" watchObservedRunningTime="2026-11-01 10:11:56.374286208 +0000 UTC m=+5989.568680764"
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.407882    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache\") pod \"d510f63d-5e04-41af-a186-0b3014af2ddf\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") "
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.407977    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs\" (UniqueName: \"kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs\") pod \"d510f63d-5e04-41af-a186-0b3014af2ddf\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") "
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.408105    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm\") pod \"d510f63d-5e04-41af-a186-0b3014af2ddf\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") "
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.408144    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7\" (UniqueName: \"kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7\") pod \"d510f63d-5e04-41af-a186-0b3014af2ddf\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") "
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.408197    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home\") pod \"d510f63d-5e04-41af-a186-0b3014af2ddf\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") "
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.408236    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir\") pod \"d510f63d-5e04-41af-a186-0b3014af2ddf\" (UID: \"d510f63d-5e04-41af-a186-0b3014af2ddf\") "
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.412664    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir" pod "d510f63d-5e04-41af-a186-0b3014af2ddf" (UID: "d510f63d-5e04-41af-a186-0b3014af2ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:11:56.417039 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.416652    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache" pod "d510f63d-5e04-41af-a186-0b3014af2ddf" (UID: "d510f63d-5e04-41af-a186-0b3014af2ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:11:56.455770 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.455654    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home" pod "d510f63d-5e04-41af-a186-0b3014af2ddf" (UID: "d510f63d-5e04-41af-a186-0b3014af2ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:11:56.606168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.505756    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs" pod "d510f63d-5e04-41af-a186-0b3014af2ddf" (UID: "d510f63d-5e04-41af-a186-0b3014af2ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:11:56.606168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.519276    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:56.606168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.519309    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:56.606168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.519323    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:56.606168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.519341    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d510f63d-5e04-41af-a186-0b3014af2ddf-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:56.606168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.528121    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7" pod "d510f63d-5e04-41af-a186-0b3014af2ddf" (UID: "d510f63d-5e04-41af-a186-0b3014af2ddf"). InnerVolumeSpecName "kube-api-access-ndps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:11:56.644564 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.643780    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm" pod "d510f63d-5e04-41af-a186-0b3014af2ddf" (UID: "d510f63d-5e04-41af-a186-0b3014af2ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:11:56.644564 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.644494    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d510f63d-5e04-41af-a186-0b3014af2ddf-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:56.644564 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.644523    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndps7\" (UniqueName: \"kubernetes.io/projected/d510f63d-5e04-41af-a186-0b3014af2ddf-kube-api-access-ndps7\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:56.745559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.744625    4314 generic.go:356] "Generic (PLEG): container finished" podID="20fb7541-2600-49b1-bab8-719215160d51" containerID="186a59595ec53762a67f4d149bac4cce42bd77a94f107656eb3136613e7f53a5" exitCode=0
Nov 01 10:11:56.745559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.744737    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c" event={"ID":"20fb7541-2600-49b1-bab8-719215160d51","Type":"ContainerDied","Data":"186a59595ec53762a67f4d149bac4cce42bd77a94f107656eb3136613e7f53a5"}
Nov 01 10:11:56.748567 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.747868    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" event={"ID":"d510f63d-5e04-41af-a186-0b3014af2ddf","Type":"ContainerDied","Data":"416b8959b93a8fb59ca4e27d012121e58a184f22b3de151a56cccbc4c196c8e9"}
Nov 01 10:11:56.748567 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.747925    4314 scope.go:122] "RemoveContainer" containerID="f6e4fa63e9f401be77a0ec772797d222daae3fcec39643475ba4fd64b3031f21"
Nov 01 10:11:56.748567 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:56.748100    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"
Nov 01 10:11:57.027587 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:57.027469    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"]
Nov 01 10:11:57.041830 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:57.041794    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z"]
Nov 01 10:11:57.079067 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:57.079006    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-46760c40-kserve-56687867fc-cd29z" podUID="d510f63d-5e04-41af-a186-0b3014af2ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.1.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:11:57.945229 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:57.945197    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:58.117113 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.117026    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc\" (UniqueName: \"kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc\") pod \"20fb7541-2600-49b1-bab8-719215160d51\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") "
Nov 01 10:11:58.117113 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.117081    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca\") pod \"20fb7541-2600-49b1-bab8-719215160d51\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") "
Nov 01 10:11:58.117383 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.117205    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca\") pod \"20fb7541-2600-49b1-bab8-719215160d51\" (UID: \"20fb7541-2600-49b1-bab8-719215160d51\") "
Nov 01 10:11:58.117445 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.117386    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca" pod "20fb7541-2600-49b1-bab8-719215160d51" (UID: "20fb7541-2600-49b1-bab8-719215160d51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:11:58.117755 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.117723    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fb7541-2600-49b1-bab8-719215160d51-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:58.119279 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.119253    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca" pod "20fb7541-2600-49b1-bab8-719215160d51" (UID: "20fb7541-2600-49b1-bab8-719215160d51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:11:58.119390 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.119342    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc" pod "20fb7541-2600-49b1-bab8-719215160d51" (UID: "20fb7541-2600-49b1-bab8-719215160d51"). InnerVolumeSpecName "kube-api-access-4q2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:11:58.218169 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.218139    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fb7541-2600-49b1-bab8-719215160d51-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:58.218169 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.218168    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q2rc\" (UniqueName: \"kubernetes.io/projected/20fb7541-2600-49b1-bab8-719215160d51-kube-api-access-4q2rc\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:11:58.790853 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.790821    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"
Nov 01 10:11:58.791157 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.790856    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c" event={"ID":"20fb7541-2600-49b1-bab8-719215160d51","Type":"ContainerDied","Data":"e61471cf01dfedf044c5ba944bf230ce0181fbb20b4ddb698a89dcc7b8580e29"}
Nov 01 10:11:58.791157 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.790905    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61471cf01dfedf044c5ba944bf230ce0181fbb20b4ddb698a89dcc7b8580e29"
Nov 01 10:11:58.927365 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:11:58.927327    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d510f63d-5e04-41af-a186-0b3014af2ddf" path="/var/lib/kubelet/pods/d510f63d-5e04-41af-a186-0b3014af2ddf/volumes"
Nov 01 10:12:09.629681 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:09.629631    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:12:09.630156 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:09.629717    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:12:11.321673 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:12:11.321629    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:12:17.182165 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:17.181639    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:12:17.182165 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:17.181720    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:12:17.471716 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:17.466635    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:12:17.471716 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:17.466711    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:12:26.046840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:26.046804    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"]
Nov 01 10:12:26.092746 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:26.092719    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e8cb47-ntqbf"]
Nov 01 10:12:26.927517 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:26.927478    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11cbc5a-2acf-4e7e-8352-7a5cc83c6231" path="/var/lib/kubelet/pods/b11cbc5a-2acf-4e7e-8352-7a5cc83c6231/volumes"
Nov 01 10:12:34.467739 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:34.467694    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:12:41.321511 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:12:41.321465    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:12:45.047418 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:45.047385    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"]
Nov 01 10:12:45.093351 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:45.093317    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa569d2-llgs6"]
Nov 01 10:12:45.476706 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:45.476628    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:12:46.926729 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:46.926697    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c3dfc9-4537-4cfa-9603-9f09d15bca26" path="/var/lib/kubelet/pods/d0c3dfc9-4537-4cfa-9603-9f09d15bca26/volumes"
Nov 01 10:12:56.462589 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.462552    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-77c6d775cc-l2l2w"]
Nov 01 10:12:56.462986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.462928    4314 cpu_manager.go:414] "containerMap: removing container" podUID="20fb7541-2600-49b1-bab8-719215160d51" containerName="revoke-keys"
Nov 01 10:12:56.462986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.462939    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fb7541-2600-49b1-bab8-719215160d51" containerName="revoke-keys"
Nov 01 10:12:56.462986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.462958    4314 cpu_manager.go:414] "containerMap: removing container" podUID="d510f63d-5e04-41af-a186-0b3014af2ddf" containerName="main"
Nov 01 10:12:56.462986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.462963    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d510f63d-5e04-41af-a186-0b3014af2ddf" containerName="main"
Nov 01 10:12:56.463181 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.463019    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d510f63d-5e04-41af-a186-0b3014af2ddf" containerName="main"
Nov 01 10:12:56.463181 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.463032    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fb7541-2600-49b1-bab8-719215160d51" containerName="revoke-keys"
Nov 01 10:12:56.466541 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.466518    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.517154 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.517129    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 10:12:56.555588 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.555556    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume\") pod \"payload-processing-77c6d775cc-l2l2w\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") " pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.555709 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.555608    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b68t\" (UniqueName: \"kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t\") pod \"payload-processing-77c6d775cc-l2l2w\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") " pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.563106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.563083    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4h9j5\""
Nov 01 10:12:56.565449 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.565433    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-77c6d775cc-l2l2w"]
Nov 01 10:12:56.656669 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.656643    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume\") pod \"payload-processing-77c6d775cc-l2l2w\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") " pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.656797 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.656682    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b68t\" (UniqueName: \"kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t\") pod \"payload-processing-77c6d775cc-l2l2w\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") " pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.657332 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.657314    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume\") pod \"payload-processing-77c6d775cc-l2l2w\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") " pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.715136 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.715073    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b68t\" (UniqueName: \"kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t\") pod \"payload-processing-77c6d775cc-l2l2w\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") " pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:56.778517 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:56.778496    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:57.142096 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:57.142073    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-77c6d775cc-l2l2w"]
Nov 01 10:12:57.144289 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:12:57.144244    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bf16a5_d08b_4b3c_af48_397f6ce3cb4f.slice/crio-6d93fe2ba5d05ffbfe1cc0ff4103c739ab4247f61554cfe808e2efdac529f683 WatchSource:0}: Error finding container 6d93fe2ba5d05ffbfe1cc0ff4103c739ab4247f61554cfe808e2efdac529f683: Status 404 returned error can't find the container with id 6d93fe2ba5d05ffbfe1cc0ff4103c739ab4247f61554cfe808e2efdac529f683
Nov 01 10:12:57.267029 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:57.267001    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w" event={"ID":"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f","Type":"ContainerStarted","Data":"a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201"}
Nov 01 10:12:57.267166 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:57.267046    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w" event={"ID":"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f","Type":"ContainerStarted","Data":"6d93fe2ba5d05ffbfe1cc0ff4103c739ab4247f61554cfe808e2efdac529f683"}
Nov 01 10:12:57.267166 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:57.267078    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:12:57.377014 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:12:57.376962    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w" podStartSLOduration=1.376948327 podStartE2EDuration="1.376948327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:12:57.376294118 +0000 UTC m=+6050.570688672" watchObservedRunningTime="2026-11-01 10:12:57.376948327 +0000 UTC m=+6050.571342881"
Nov 01 10:13:08.275595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:08.275562    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:13:10.663726 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.663695    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-797f57964-nlqd8"]
Nov 01 10:13:10.667322 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.667302    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.721778 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.721750    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-797f57964-nlqd8"]
Nov 01 10:13:10.781321 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.781290    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3415ad8e-a2dd-4804-9eff-7145c8fa8678-plugins-config-volume\") pod \"payload-processing-797f57964-nlqd8\" (UID: \"3415ad8e-a2dd-4804-9eff-7145c8fa8678\") " pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.781461 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.781359    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkjb\" (UniqueName: \"kubernetes.io/projected/3415ad8e-a2dd-4804-9eff-7145c8fa8678-kube-api-access-xrkjb\") pod \"payload-processing-797f57964-nlqd8\" (UID: \"3415ad8e-a2dd-4804-9eff-7145c8fa8678\") " pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.882395 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.882373    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3415ad8e-a2dd-4804-9eff-7145c8fa8678-plugins-config-volume\") pod \"payload-processing-797f57964-nlqd8\" (UID: \"3415ad8e-a2dd-4804-9eff-7145c8fa8678\") " pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.882522 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.882431    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkjb\" (UniqueName: \"kubernetes.io/projected/3415ad8e-a2dd-4804-9eff-7145c8fa8678-kube-api-access-xrkjb\") pod \"payload-processing-797f57964-nlqd8\" (UID: \"3415ad8e-a2dd-4804-9eff-7145c8fa8678\") " pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.883076 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.883058    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3415ad8e-a2dd-4804-9eff-7145c8fa8678-plugins-config-volume\") pod \"payload-processing-797f57964-nlqd8\" (UID: \"3415ad8e-a2dd-4804-9eff-7145c8fa8678\") " pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.940334 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.940264    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkjb\" (UniqueName: \"kubernetes.io/projected/3415ad8e-a2dd-4804-9eff-7145c8fa8678-kube-api-access-xrkjb\") pod \"payload-processing-797f57964-nlqd8\" (UID: \"3415ad8e-a2dd-4804-9eff-7145c8fa8678\") " pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:10.979671 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:10.979644    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:11.321491 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:13:11.321458    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:13:11.341815 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:11.341792    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-797f57964-nlqd8"]
Nov 01 10:13:11.344427 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:13:11.344398    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3415ad8e_a2dd_4804_9eff_7145c8fa8678.slice/crio-b69c722090cde91ae9457910885e9d010b1aa9a9776b1d61cd0344c6d423e610 WatchSource:0}: Error finding container b69c722090cde91ae9457910885e9d010b1aa9a9776b1d61cd0344c6d423e610: Status 404 returned error can't find the container with id b69c722090cde91ae9457910885e9d010b1aa9a9776b1d61cd0344c6d423e610
Nov 01 10:13:12.349600 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:12.349568    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-797f57964-nlqd8" event={"ID":"3415ad8e-a2dd-4804-9eff-7145c8fa8678","Type":"ContainerStarted","Data":"815a2aec38f26cc47070da4db5ae5896d4aea34961bec731290f5f65f98d2146"}
Nov 01 10:13:12.350092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:12.349624    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-797f57964-nlqd8" event={"ID":"3415ad8e-a2dd-4804-9eff-7145c8fa8678","Type":"ContainerStarted","Data":"b69c722090cde91ae9457910885e9d010b1aa9a9776b1d61cd0344c6d423e610"}
Nov 01 10:13:12.350092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:12.349651    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:12.462229 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:12.462173    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-797f57964-nlqd8" podStartSLOduration=2.462158797 podStartE2EDuration="2.462158797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:13:12.458833598 +0000 UTC m=+6065.653228151" watchObservedRunningTime="2026-11-01 10:13:12.462158797 +0000 UTC m=+6065.656553351"
Nov 01 10:13:23.358466 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:23.358429    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-797f57964-nlqd8"
Nov 01 10:13:23.515572 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:23.515515    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-77c6d775cc-l2l2w"]
Nov 01 10:13:26.311106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:26.311072    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:13:26.716143 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:26.716059    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:13:28.516467 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.516408    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w" podUID="56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" containerName="payload-processing" containerID="cri-o://a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201" gracePeriod=25
Nov 01 10:13:28.761875 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.761849    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:13:28.851385 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.851311    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t\" (UniqueName: \"kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t\") pod \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") "
Nov 01 10:13:28.851385 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.851353    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume\") pod \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\" (UID: \"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f\") "
Nov 01 10:13:28.851765 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.851736    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume" pod "56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" (UID: "56bf16a5-d08b-4b3c-af48-397f6ce3cb4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:13:28.853453 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.853427    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t" pod "56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" (UID: "56bf16a5-d08b-4b3c-af48-397f6ce3cb4f"). InnerVolumeSpecName "kube-api-access-9b68t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:13:28.952316 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.952286    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b68t\" (UniqueName: \"kubernetes.io/projected/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-kube-api-access-9b68t\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:13:28.952316 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:28.952313    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:13:29.431190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.431157    4314 generic.go:356] "Generic (PLEG): container finished" podID="56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" containerID="a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201" exitCode=0
Nov 01 10:13:29.431371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.431201    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w"
Nov 01 10:13:29.431371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.431233    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w" event={"ID":"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f","Type":"ContainerDied","Data":"a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201"}
Nov 01 10:13:29.431371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.431281    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77c6d775cc-l2l2w" event={"ID":"56bf16a5-d08b-4b3c-af48-397f6ce3cb4f","Type":"ContainerDied","Data":"6d93fe2ba5d05ffbfe1cc0ff4103c739ab4247f61554cfe808e2efdac529f683"}
Nov 01 10:13:29.431371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.431304    4314 scope.go:122] "RemoveContainer" containerID="a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201"
Nov 01 10:13:29.440046 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.440030    4314 scope.go:122] "RemoveContainer" containerID="a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201"
Nov 01 10:13:29.440310 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:13:29.440293    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201\": container with ID starting with a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201 not found: ID does not exist" containerID="a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201"
Nov 01 10:13:29.440380 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.440321    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201"} err="failed to get container status \"a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201\": rpc error: code = NotFound desc = could not find container \"a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201\": container with ID starting with a5c34d65bcfdbc4d5c1e0bd88f509c437008d95371981deb1cb1bc87949fe201 not found: ID does not exist"
Nov 01 10:13:29.459706 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.459681    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:13:29.585072 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.585043    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-77c6d775cc-l2l2w"]
Nov 01 10:13:29.631270 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:29.631239    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-77c6d775cc-l2l2w"]
Nov 01 10:13:30.927709 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:30.927676    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" path="/var/lib/kubelet/pods/56bf16a5-d08b-4b3c-af48-397f6ce3cb4f/volumes"
Nov 01 10:13:34.803557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:34.802618    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:13:34.803557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:34.802680    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:13:34.878004 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:13:34.877914    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:13:41.321560 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:13:41.321503    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:14:11.323083 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:14:11.323045    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:14:11.702610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:14:11.702486    4314 scope.go:122] "RemoveContainer" containerID="4b64f050d7e09cc8fa4379256c401b615f2d1867f599a30730210127054cd79d"
Nov 01 10:14:11.711443 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:14:11.711418    4314 scope.go:122] "RemoveContainer" containerID="dfb2c21cc5a47457d1dad7e4596985908c1871a41567d5626cdf6914d9102d47"
Nov 01 10:14:35.204561 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:14:35.204495    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:14:35.357679 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:14:35.357646    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:14:41.321894 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:14:41.321851    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:15:11.322010 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:15:11.321975    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:15:30.639663 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:30.639631    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:15:30.796010 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:30.795977    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:15:31.213508 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:31.213472    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:15:35.572961 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:35.572925    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:15:37.370136 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:37.370103    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:15:41.321652 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:15:41.321617    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:15:47.181559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:47.180648    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:15:47.181559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:47.180726    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:15:47.456845 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:47.456742    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:15:47.457145 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:15:47.457114    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:16:11.466070 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:16:11.466027    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:16:31.046056 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:31.045969    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"]
Nov 01 10:16:31.093625 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:31.093583    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zxfw2"]
Nov 01 10:16:32.927432 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:32.927400    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3d8a18-72b4-4158-8869-a6dc0141e657" path="/var/lib/kubelet/pods/7e3d8a18-72b4-4158-8869-a6dc0141e657/volumes"
Nov 01 10:16:36.042708 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:36.042672    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:16:41.305610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:41.305572    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:16:41.322661 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:16:41.322625    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:16:41.466734 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:41.466700    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:16:47.185557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:47.182646    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:16:47.185557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:47.182730    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:16:47.453787 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:47.453689    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:16:47.454092 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:47.454038    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:16:58.048563 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:58.048519    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"]
Nov 01 10:16:58.094238 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:58.094208    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2qk7c"]
Nov 01 10:16:58.927374 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:16:58.927339    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fb7541-2600-49b1-bab8-719215160d51" path="/var/lib/kubelet/pods/20fb7541-2600-49b1-bab8-719215160d51/volumes"
Nov 01 10:17:11.321700 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:17:11.321660    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:17:34.805216 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:17:34.805159    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:17:34.805962 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:17:34.805911    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:17:36.342637 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:17:36.342584    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:17:41.322391 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:17:41.322350    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:18:09.642628 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:09.641443    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:18:09.642628 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:09.641513    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:18:11.324096 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:18:11.324055    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:18:11.787832 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:11.787800    4314 scope.go:122] "RemoveContainer" containerID="d73a5ded0c142d81347e7732212b3b48312178ecce5b6a643d04fcc8d790bb40"
Nov 01 10:18:11.797265 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:11.797238    4314 scope.go:122] "RemoveContainer" containerID="186a59595ec53762a67f4d149bac4cce42bd77a94f107656eb3136613e7f53a5"
Nov 01 10:18:36.732417 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:36.732377    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:18:41.323470 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:18:41.323431    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:18:47.467936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:47.467790    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:18:47.469189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:18:47.469143    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:19:11.321882 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:19:11.321847    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:19:19.826611 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:19:19.825655    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:19:19.826611 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:19:19.825755    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:19:37.036993 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:19:37.036953    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:19:41.322168 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:19:41.322131    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:20:11.321784 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:20:11.321741    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:20:17.181596 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:17.179707    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:20:17.181596 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:17.179774    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:20:17.449752 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:17.449658    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:20:17.450055 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:17.450012    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:20:37.388257 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:37.388222    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:41.322240 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:20:41.322200    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:20:44.127388 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:44.127351    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:44.763591 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:44.762909    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:45.231617 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:45.210065    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:49.133964 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:49.133889    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:49.294230 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:49.294198    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:49.452117 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:49.452034    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:52.355155 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.355116    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:52.525423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.525388    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:52.776158 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.776119    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"]
Nov 01 10:20:52.776668 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.776649    4314 cpu_manager.go:414] "containerMap: removing container" podUID="56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" containerName="payload-processing"
Nov 01 10:20:52.776776 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.776668    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" containerName="payload-processing"
Nov 01 10:20:52.776776 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.776754    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bf16a5-d08b-4b3c-af48-397f6ce3cb4f" containerName="payload-processing"
Nov 01 10:20:52.779932 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.779912    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:52.880129 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.878657    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-23b415\""
Nov 01 10:20:52.880129 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.878754    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e45b1a16-dockercfg-lgfjk\""
Nov 01 10:20:52.880129 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.880064    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"]
Nov 01 10:20:52.896729 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.896686    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-23b415-8459f48896\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:52.896905 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.896818    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcns8\" (UniqueName: \"kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8\") pod \"payload-pre-processing-e2e-worker-single-main-23b415-8459f48896\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:52.998046 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.998003    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-23b415-8459f48896\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:52.998501 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.998468    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcns8\" (UniqueName: \"kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8\") pod \"payload-pre-processing-e2e-worker-single-main-23b415-8459f48896\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:52.998923 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:52.998901    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-23b415-8459f48896\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:53.056040 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:53.055957    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcns8\" (UniqueName: \"kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8\") pod \"payload-pre-processing-e2e-worker-single-main-23b415-8459f48896\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:53.092766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:53.092733    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:53.457585 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:53.457559    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"]
Nov 01 10:20:53.459984 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:20:53.459956    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17300ed_530a_4cd7_a545_2b9dabff5091.slice/crio-c81331404e2dc5de05338aaffd1046e81f9c4ca4880b437e7146347cb9472ea1 WatchSource:0}: Error finding container c81331404e2dc5de05338aaffd1046e81f9c4ca4880b437e7146347cb9472ea1: Status 404 returned error can't find the container with id c81331404e2dc5de05338aaffd1046e81f9c4ca4880b437e7146347cb9472ea1
Nov 01 10:20:54.420754 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:54.420719    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" event={"ID":"f17300ed-530a-4cd7-a545-2b9dabff5091","Type":"ContainerStarted","Data":"44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4"}
Nov 01 10:20:54.420940 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:54.420778    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" event={"ID":"f17300ed-530a-4cd7-a545-2b9dabff5091","Type":"ContainerStarted","Data":"c81331404e2dc5de05338aaffd1046e81f9c4ca4880b437e7146347cb9472ea1"}
Nov 01 10:20:54.420940 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:54.420806    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:20:54.531798 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:54.531746    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" podStartSLOduration=2.53173206 podStartE2EDuration="2.53173206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:20:54.528603506 +0000 UTC m=+6527.722998063" watchObservedRunningTime="2026-11-01 10:20:54.53173206 +0000 UTC m=+6527.726126614"
Nov 01 10:20:55.296894 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:55.296851    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:55.441837 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:55.441800    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:55.590789 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:55.590710    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:20:59.967023 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:20:59.966983    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:04.632440 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:04.632401    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:05.429283 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:05.429254    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:21:06.545374 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:06.545340    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:09.157685 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:09.157650    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:11.322295 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:21:11.322258    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:21:14.017701 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:14.017670    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:14.174036 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:14.174007    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:14.643045 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:14.642975    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:14.806355 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:14.806302    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:17.214034 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.213999    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"]
Nov 01 10:21:17.217644 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.217622    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.313223 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.313190    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-23b415\"/\"default-dockercfg-tvnmg\""
Nov 01 10:21:17.313401 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.313250    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-23b415\"/\"openshift-service-ca.crt\""
Nov 01 10:21:17.313401 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.313250    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-23b415\"/\"kube-root-ca.crt\""
Nov 01 10:21:17.315403 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.315380    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"]
Nov 01 10:21:17.338514 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.338486    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:17.354752 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.354722    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.354888 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.354766    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.354888 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.354875    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.354968 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.354896    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5xt\" (UniqueName: \"kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.354968 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.354941    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.355077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.355059    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.367890 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.367858    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-23b415\"/\"simulator-single-main-23b415-kserve-self-signed-certs\""
Nov 01 10:21:17.456604 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.456568    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.456819 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.456612    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.456819 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.456659    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.456963 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.456816    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5xt\" (UniqueName: \"kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.456963 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.456907    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.457106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.456966    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.457106 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.457033    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.457306 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.457281    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.457471 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.457437    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.459119 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.459097    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.459359 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.459338    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.515143 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.515103    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5xt\" (UniqueName: \"kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt\") pod \"simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") " pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.529033 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.529001    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:17.899455 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:17.899426    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"]
Nov 01 10:21:17.901273 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:21:17.901244    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3084d160_6b2b_4798_aa96_c7046f3ac3d1.slice/crio-c08528bc225fe559c740fa43e38aa40131ded051849a16f3145ecd2d09a355c2 WatchSource:0}: Error finding container c08528bc225fe559c740fa43e38aa40131ded051849a16f3145ecd2d09a355c2: Status 404 returned error can't find the container with id c08528bc225fe559c740fa43e38aa40131ded051849a16f3145ecd2d09a355c2
Nov 01 10:21:18.541352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:18.541315    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx" event={"ID":"3084d160-6b2b-4798-aa96-c7046f3ac3d1","Type":"ContainerStarted","Data":"a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c"}
Nov 01 10:21:18.541897 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:18.541368    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx" event={"ID":"3084d160-6b2b-4798-aa96-c7046f3ac3d1","Type":"ContainerStarted","Data":"c08528bc225fe559c740fa43e38aa40131ded051849a16f3145ecd2d09a355c2"}
Nov 01 10:21:18.650875 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:18.650818    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx" podStartSLOduration=1.650801374 podStartE2EDuration="1.650801374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:21:18.650063614 +0000 UTC m=+6551.844458168" watchObservedRunningTime="2026-11-01 10:21:18.650801374 +0000 UTC m=+6551.845195930"
Nov 01 10:21:18.675762 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:18.675727    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:19.530063 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:19.530029    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:23.349922 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:23.349888    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:27.530215 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:27.530173    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:27.542405 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:27.542378    4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:27.596995 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:27.596964    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:21:27.860357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:27.860248    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:32.459410 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:32.459372    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:37.059437 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:37.059404    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:37.735347 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:37.735308    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:41.322603 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:21:41.322566    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:21:41.656784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:41.656701    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:46.238368 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:46.238332    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:50.855330 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:50.855298    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:55.441800 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:55.441759    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:59.127224 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.127185    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:21:59.235458 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.235419    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"]
Nov 01 10:21:59.239342 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.239325    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.336604 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.336567    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:21:59.336604 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.336564    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:21:59.336846 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.336570    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:21:59.336846 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.336670    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:21:59.338900 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.338878    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"]
Nov 01 10:21:59.364601 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.364577    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.364757 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.364634    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfr5k\" (UniqueName: \"kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.364757 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.364735    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.466030 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.465940    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfr5k\" (UniqueName: \"kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.466030 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.465995    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.466284 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.466067    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.466667 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.466645    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.468417 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.468391    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.523899 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.523871    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfr5k\" (UniqueName: \"kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k\") pod \"maas-api-revoke-sub-10603c16-q99cx\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.553014 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.552990    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:21:59.922639 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:21:59.922611    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"]
Nov 01 10:21:59.924984 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:21:59.924954    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b429c34_418d_45c1_8714_dbcddd571243.slice/crio-43ba6585433b216c54309db72e488da1ba9d22d0771ee320b2098b71cb4d9f39 WatchSource:0}: Error finding container 43ba6585433b216c54309db72e488da1ba9d22d0771ee320b2098b71cb4d9f39: Status 404 returned error can't find the container with id 43ba6585433b216c54309db72e488da1ba9d22d0771ee320b2098b71cb4d9f39
Nov 01 10:22:00.069807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:00.069778    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:00.757984 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:00.757949    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx" event={"ID":"5b429c34-418d-45c1-8714-dbcddd571243","Type":"ContainerStarted","Data":"3387f939c5427b567be89ed73de6ed5e26f7368ce705756325a8c1f05208e0f8"}
Nov 01 10:22:00.758384 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:00.758038    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx" event={"ID":"5b429c34-418d-45c1-8714-dbcddd571243","Type":"ContainerStarted","Data":"43ba6585433b216c54309db72e488da1ba9d22d0771ee320b2098b71cb4d9f39"}
Nov 01 10:22:00.867791 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:00.867733    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx" podStartSLOduration=1.867718032 podStartE2EDuration="1.867718032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:22:00.86689723 +0000 UTC m=+6594.061291787" watchObservedRunningTime="2026-11-01 10:22:00.867718032 +0000 UTC m=+6594.062112587"
Nov 01 10:22:01.764374 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:01.764344    4314 generic.go:356] "Generic (PLEG): container finished" podID="5b429c34-418d-45c1-8714-dbcddd571243" containerID="3387f939c5427b567be89ed73de6ed5e26f7368ce705756325a8c1f05208e0f8" exitCode=0
Nov 01 10:22:01.764830 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:01.764430    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx" event={"ID":"5b429c34-418d-45c1-8714-dbcddd571243","Type":"ContainerDied","Data":"3387f939c5427b567be89ed73de6ed5e26f7368ce705756325a8c1f05208e0f8"}
Nov 01 10:22:02.900184 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:02.900162    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:22:03.000667 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.000642    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca\") pod \"5b429c34-418d-45c1-8714-dbcddd571243\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") "
Nov 01 10:22:03.000810 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.000743    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca\") pod \"5b429c34-418d-45c1-8714-dbcddd571243\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") "
Nov 01 10:22:03.000810 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.000790    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k\" (UniqueName: \"kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k\") pod \"5b429c34-418d-45c1-8714-dbcddd571243\" (UID: \"5b429c34-418d-45c1-8714-dbcddd571243\") "
Nov 01 10:22:03.000979 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.000952    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca" pod "5b429c34-418d-45c1-8714-dbcddd571243" (UID: "5b429c34-418d-45c1-8714-dbcddd571243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:22:03.001224 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.001201    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:03.002831 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.002810    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k" pod "5b429c34-418d-45c1-8714-dbcddd571243" (UID: "5b429c34-418d-45c1-8714-dbcddd571243"). InnerVolumeSpecName "kube-api-access-kfr5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:22:03.002989 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.002973    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca" pod "5b429c34-418d-45c1-8714-dbcddd571243" (UID: "5b429c34-418d-45c1-8714-dbcddd571243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:22:03.102287 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.102208    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b429c34-418d-45c1-8714-dbcddd571243-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:03.102287 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.102232    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfr5k\" (UniqueName: \"kubernetes.io/projected/5b429c34-418d-45c1-8714-dbcddd571243-kube-api-access-kfr5k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:03.775289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.775251    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx" event={"ID":"5b429c34-418d-45c1-8714-dbcddd571243","Type":"ContainerDied","Data":"43ba6585433b216c54309db72e488da1ba9d22d0771ee320b2098b71cb4d9f39"}
Nov 01 10:22:03.775289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.775289    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"
Nov 01 10:22:03.775507 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:03.775306    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ba6585433b216c54309db72e488da1ba9d22d0771ee320b2098b71cb4d9f39"
Nov 01 10:22:04.677500 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:04.677454    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:09.253339 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:09.253306    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:10.923948 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:10.923904    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:11.321497 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:22:11.321463    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:22:11.401883 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:11.401852    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:13.853325 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:13.853276    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:15.871880 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:15.871841    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"]
Nov 01 10:22:15.971077 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:15.971042    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:17.897333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:17.897294    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:18.052130 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:18.052100    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:18.308693 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:18.308660    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"]
Nov 01 10:22:18.428871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:18.428836    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:18.644595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:18.644495    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:23.309574 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:23.309520    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerName="payload-pre-processing" containerID="cri-o://44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4" gracePeriod=25
Nov 01 10:22:23.753825 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:23.753789    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:23.904744 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:23.904713    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:25.427808 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:25.427767    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:22:30.892367 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:30.892308    4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx" podUID="3084d160-6b2b-4798-aa96-c7046f3ac3d1" containerName="main" containerID="cri-o://a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c" gracePeriod=45
Nov 01 10:22:31.140503 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.140479    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:22:31.276206 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276172    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt\" (UniqueName: \"kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt\") pod \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") "
Nov 01 10:22:31.276407 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276269    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir\") pod \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") "
Nov 01 10:22:31.276407 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276347    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm\") pod \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") "
Nov 01 10:22:31.276556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276419    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs\") pod \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") "
Nov 01 10:22:31.276556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276461    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache\") pod \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") "
Nov 01 10:22:31.276556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276490    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home\") pod \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\" (UID: \"3084d160-6b2b-4798-aa96-c7046f3ac3d1\") "
Nov 01 10:22:31.276751 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276583    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir" pod "3084d160-6b2b-4798-aa96-c7046f3ac3d1" (UID: "3084d160-6b2b-4798-aa96-c7046f3ac3d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:22:31.277059 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276871    4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tmp-dir\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:31.277059 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.276982    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache" pod "3084d160-6b2b-4798-aa96-c7046f3ac3d1" (UID: "3084d160-6b2b-4798-aa96-c7046f3ac3d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:22:31.277256 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.277107    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home" pod "3084d160-6b2b-4798-aa96-c7046f3ac3d1" (UID: "3084d160-6b2b-4798-aa96-c7046f3ac3d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:22:31.278738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.278712    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm" pod "3084d160-6b2b-4798-aa96-c7046f3ac3d1" (UID: "3084d160-6b2b-4798-aa96-c7046f3ac3d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:22:31.278877 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.278858    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt" pod "3084d160-6b2b-4798-aa96-c7046f3ac3d1" (UID: "3084d160-6b2b-4798-aa96-c7046f3ac3d1"). InnerVolumeSpecName "kube-api-access-xn5xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:22:31.278974 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.278951    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs" pod "3084d160-6b2b-4798-aa96-c7046f3ac3d1" (UID: "3084d160-6b2b-4798-aa96-c7046f3ac3d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:22:31.377976 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.377941    4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-dshm\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:31.377976 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.377970    4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3084d160-6b2b-4798-aa96-c7046f3ac3d1-tls-certs\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:31.377976 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.377980    4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-model-cache\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:31.378230 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.377988    4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3084d160-6b2b-4798-aa96-c7046f3ac3d1-home\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:31.378230 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.377999    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5xt\" (UniqueName: \"kubernetes.io/projected/3084d160-6b2b-4798-aa96-c7046f3ac3d1-kube-api-access-xn5xt\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:31.919959 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.919927    4314 generic.go:356] "Generic (PLEG): container finished" podID="3084d160-6b2b-4798-aa96-c7046f3ac3d1" containerID="a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c" exitCode=0
Nov 01 10:22:31.920471 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.919977    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"
Nov 01 10:22:31.920471 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.920007    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx" event={"ID":"3084d160-6b2b-4798-aa96-c7046f3ac3d1","Type":"ContainerDied","Data":"a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c"}
Nov 01 10:22:31.920471 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.920053    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx" event={"ID":"3084d160-6b2b-4798-aa96-c7046f3ac3d1","Type":"ContainerDied","Data":"c08528bc225fe559c740fa43e38aa40131ded051849a16f3145ecd2d09a355c2"}
Nov 01 10:22:31.920471 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.920071    4314 scope.go:122] "RemoveContainer" containerID="a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c"
Nov 01 10:22:31.929780 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.929766    4314 scope.go:122] "RemoveContainer" containerID="a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c"
Nov 01 10:22:31.930031 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:22:31.930013    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c\": container with ID starting with a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c not found: ID does not exist" containerID="a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c"
Nov 01 10:22:31.930079 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:31.930040    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c"} err="failed to get container status \"a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c\": rpc error: code = NotFound desc = could not find container \"a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c\": container with ID starting with a546b927b41f03ee08436cf13c835f0a9be5b37926496805db27af267728949c not found: ID does not exist"
Nov 01 10:22:32.074830 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:32.074798    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"]
Nov 01 10:22:32.120703 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:32.120669    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-23b415/simulator-single-main-23b415-kserve-5ff4d6b65-fq8lx"]
Nov 01 10:22:32.927292 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:32.927258    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3084d160-6b2b-4798-aa96-c7046f3ac3d1" path="/var/lib/kubelet/pods/3084d160-6b2b-4798-aa96-c7046f3ac3d1/volumes"
Nov 01 10:22:35.425737 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:35.425685    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:35.427403 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:35.427376    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 10:22:35.600586 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:35.600543    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:38.090111 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:38.090068    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:39.684768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:39.684734    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:41.322029 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:22:41.321993    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:22:41.679787 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:41.679708    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:41.865224 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:41.865189    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:22:43.560090 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.560057    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:22:43.590735 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.590664    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8\" (UniqueName: \"kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8\") pod \"f17300ed-530a-4cd7-a545-2b9dabff5091\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") "
Nov 01 10:22:43.590876 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.590759    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume\") pod \"f17300ed-530a-4cd7-a545-2b9dabff5091\" (UID: \"f17300ed-530a-4cd7-a545-2b9dabff5091\") "
Nov 01 10:22:43.591112 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.591089    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume" pod "f17300ed-530a-4cd7-a545-2b9dabff5091" (UID: "f17300ed-530a-4cd7-a545-2b9dabff5091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:22:43.592906 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.592882    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8" pod "f17300ed-530a-4cd7-a545-2b9dabff5091" (UID: "f17300ed-530a-4cd7-a545-2b9dabff5091"). InnerVolumeSpecName "kube-api-access-hcns8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:22:43.691469 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.691432    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcns8\" (UniqueName: \"kubernetes.io/projected/f17300ed-530a-4cd7-a545-2b9dabff5091-kube-api-access-hcns8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:43.691469 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.691466    4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17300ed-530a-4cd7-a545-2b9dabff5091-plugins-config-volume\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:22:43.986173 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.986144    4314 generic.go:356] "Generic (PLEG): container finished" podID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerID="44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4" exitCode=0
Nov 01 10:22:43.986342 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.986187    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"
Nov 01 10:22:43.986342 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.986185    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" event={"ID":"f17300ed-530a-4cd7-a545-2b9dabff5091","Type":"ContainerDied","Data":"44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4"}
Nov 01 10:22:43.986342 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.986319    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896" event={"ID":"f17300ed-530a-4cd7-a545-2b9dabff5091","Type":"ContainerDied","Data":"c81331404e2dc5de05338aaffd1046e81f9c4ca4880b437e7146347cb9472ea1"}
Nov 01 10:22:43.986574 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.986355    4314 scope.go:122] "RemoveContainer" containerID="44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4"
Nov 01 10:22:43.995051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.995036    4314 scope.go:122] "RemoveContainer" containerID="44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4"
Nov 01 10:22:43.995316 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:22:43.995297    4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4\": container with ID starting with 44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4 not found: ID does not exist" containerID="44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4"
Nov 01 10:22:43.995371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:43.995324    4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4"} err="failed to get container status \"44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4\": rpc error: code = NotFound desc = could not find container \"44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4\": container with ID starting with 44781bdd106552e6e142906f3e60637503320c0486a7b897bfe54a48fb36ebf4 not found: ID does not exist"
Nov 01 10:22:44.140632 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:44.140601    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"]
Nov 01 10:22:44.142807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:44.142778    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-23b415-8459f48896"]
Nov 01 10:22:44.927049 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:22:44.927014    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" path="/var/lib/kubelet/pods/f17300ed-530a-4cd7-a545-2b9dabff5091/volumes"
Nov 01 10:23:11.323099 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:23:11.323063    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:23:38.551000 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:23:38.550965    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:23:39.507242 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:23:39.507196    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:23:41.321640 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:23:41.321600    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:24:11.321733 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:24:11.321649    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:24:38.786516 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:38.786466    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:24:41.322056 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:24:41.322018    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:24:47.180717 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:47.180676    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:24:47.181486 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:47.181439    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:24:47.456827 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:47.456689    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:24:47.457256 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:47.457225    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:24:54.633098 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:54.632625    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:24:54.633098 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:24:54.632700    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:25:11.321719 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:25:11.321679    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:25:22.708884 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:22.708791    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:25:22.861756 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:22.861720    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:25:30.767225 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767188    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"]
Nov 01 10:25:30.767658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767611    4314 cpu_manager.go:414] "containerMap: removing container" podUID="3084d160-6b2b-4798-aa96-c7046f3ac3d1" containerName="main"
Nov 01 10:25:30.767658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767622    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3084d160-6b2b-4798-aa96-c7046f3ac3d1" containerName="main"
Nov 01 10:25:30.767658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767641    4314 cpu_manager.go:414] "containerMap: removing container" podUID="5b429c34-418d-45c1-8714-dbcddd571243" containerName="revoke-keys"
Nov 01 10:25:30.767658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767646    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b429c34-418d-45c1-8714-dbcddd571243" containerName="revoke-keys"
Nov 01 10:25:30.767658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767656    4314 cpu_manager.go:414] "containerMap: removing container" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerName="payload-pre-processing"
Nov 01 10:25:30.767658 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767662    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerName="payload-pre-processing"
Nov 01 10:25:30.767883 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767726    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b429c34-418d-45c1-8714-dbcddd571243" containerName="revoke-keys"
Nov 01 10:25:30.767883 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767734    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17300ed-530a-4cd7-a545-2b9dabff5091" containerName="payload-pre-processing"
Nov 01 10:25:30.767883 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.767748    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3084d160-6b2b-4798-aa96-c7046f3ac3d1" containerName="main"
Nov 01 10:25:30.770942 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.770918    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.825598 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.825566    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"]
Nov 01 10:25:30.868684 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.868648    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:25:30.868684 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.868649    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:25:30.868684 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.868649    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:25:30.868915 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.868705    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:25:30.885766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.885737    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.885906 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.885800    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.885964 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.885913    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dncp\" (UniqueName: \"kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.987192 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.987149    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.987371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.987244    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.987371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.987278    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dncp\" (UniqueName: \"kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.987885 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.987858    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:30.989686 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:30.989668    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:31.044595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:31.044510    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dncp\" (UniqueName: \"kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp\") pod \"maas-api-revoke-sub-f7abfbdf-wkgfl\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:31.084003 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:31.083978    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:31.452722 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:31.452698    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"]
Nov 01 10:25:31.454784 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:25:31.454762    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fdadd0_0371_4f60_b22c_74e711307199.slice/crio-1e5979450cbcf8aea79229c938b9d6ba90920eb10d379e01b7649ea5c54b23c1 WatchSource:0}: Error finding container 1e5979450cbcf8aea79229c938b9d6ba90920eb10d379e01b7649ea5c54b23c1: Status 404 returned error can't find the container with id 1e5979450cbcf8aea79229c938b9d6ba90920eb10d379e01b7649ea5c54b23c1
Nov 01 10:25:32.019473 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:32.019377    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl" event={"ID":"69fdadd0-0371-4f60-b22c-74e711307199","Type":"ContainerStarted","Data":"9068ff7bb6ebaa9bc2ae08a0a26468dcd678dc30cee969212afdac70773ae272"}
Nov 01 10:25:32.019473 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:32.019443    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl" event={"ID":"69fdadd0-0371-4f60-b22c-74e711307199","Type":"ContainerStarted","Data":"1e5979450cbcf8aea79229c938b9d6ba90920eb10d379e01b7649ea5c54b23c1"}
Nov 01 10:25:32.129296 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:32.129243    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl" podStartSLOduration=2.12922945 podStartE2EDuration="2.12922945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:25:32.12847327 +0000 UTC m=+6805.322867825" watchObservedRunningTime="2026-11-01 10:25:32.12922945 +0000 UTC m=+6805.323624005"
Nov 01 10:25:33.025909 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:33.025879    4314 generic.go:356] "Generic (PLEG): container finished" podID="69fdadd0-0371-4f60-b22c-74e711307199" containerID="9068ff7bb6ebaa9bc2ae08a0a26468dcd678dc30cee969212afdac70773ae272" exitCode=0
Nov 01 10:25:33.026281 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:33.025959    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl" event={"ID":"69fdadd0-0371-4f60-b22c-74e711307199","Type":"ContainerDied","Data":"9068ff7bb6ebaa9bc2ae08a0a26468dcd678dc30cee969212afdac70773ae272"}
Nov 01 10:25:34.159211 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.159187    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:34.318108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.318039    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca\") pod \"69fdadd0-0371-4f60-b22c-74e711307199\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") "
Nov 01 10:25:34.318254 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.318180    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp\" (UniqueName: \"kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp\") pod \"69fdadd0-0371-4f60-b22c-74e711307199\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") "
Nov 01 10:25:34.318254 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.318224    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca\") pod \"69fdadd0-0371-4f60-b22c-74e711307199\" (UID: \"69fdadd0-0371-4f60-b22c-74e711307199\") "
Nov 01 10:25:34.318381 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.318360    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca" pod "69fdadd0-0371-4f60-b22c-74e711307199" (UID: "69fdadd0-0371-4f60-b22c-74e711307199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:25:34.318562 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.318514    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:34.320255 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.320235    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca" pod "69fdadd0-0371-4f60-b22c-74e711307199" (UID: "69fdadd0-0371-4f60-b22c-74e711307199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:25:34.320360 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.320347    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp" pod "69fdadd0-0371-4f60-b22c-74e711307199" (UID: "69fdadd0-0371-4f60-b22c-74e711307199"). InnerVolumeSpecName "kube-api-access-7dncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:25:34.419048 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.419021    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dncp\" (UniqueName: \"kubernetes.io/projected/69fdadd0-0371-4f60-b22c-74e711307199-kube-api-access-7dncp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:34.419048 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:34.419045    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fdadd0-0371-4f60-b22c-74e711307199-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:35.037322 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:35.037292    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"
Nov 01 10:25:35.037494 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:35.037292    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl" event={"ID":"69fdadd0-0371-4f60-b22c-74e711307199","Type":"ContainerDied","Data":"1e5979450cbcf8aea79229c938b9d6ba90920eb10d379e01b7649ea5c54b23c1"}
Nov 01 10:25:35.037494 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:35.037438    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5979450cbcf8aea79229c938b9d6ba90920eb10d379e01b7649ea5c54b23c1"
Nov 01 10:25:39.146433 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:39.146403    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:25:41.322896 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:25:41.322858    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:25:41.513132 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.513098    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"]
Nov 01 10:25:41.513515 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.513502    4314 cpu_manager.go:414] "containerMap: removing container" podUID="69fdadd0-0371-4f60-b22c-74e711307199" containerName="revoke-keys"
Nov 01 10:25:41.513599 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.513516    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fdadd0-0371-4f60-b22c-74e711307199" containerName="revoke-keys"
Nov 01 10:25:41.513647 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.513627    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fdadd0-0371-4f60-b22c-74e711307199" containerName="revoke-keys"
Nov 01 10:25:41.516981 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.516949    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.581261 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.581179    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.581432 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.581410    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.581493 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.581446    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xq6s\" (UniqueName: \"kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.614618 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.614594    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:25:41.614766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.614683    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:25:41.614766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.614683    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:25:41.614766 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.614735    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:25:41.617079 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.617061    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"]
Nov 01 10:25:41.682332 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.682299    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.682332 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.682335    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xq6s\" (UniqueName: \"kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.682621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.682387    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.683007 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.682984    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.684729 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.684707    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.740348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.740299    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xq6s\" (UniqueName: \"kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s\") pod \"maas-api-revoke-sub-65e0baa9-g4v5n\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:41.829404 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:41.829375    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:42.200499 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:42.200464    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"]
Nov 01 10:25:42.203069 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:25:42.203045    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500d094f_6f86_46ec_8869_8372d772e308.slice/crio-34e254169529e5d3f1b7ce8c93030e958ca43f8420e4f2578c081ae8cb55ec31 WatchSource:0}: Error finding container 34e254169529e5d3f1b7ce8c93030e958ca43f8420e4f2578c081ae8cb55ec31: Status 404 returned error can't find the container with id 34e254169529e5d3f1b7ce8c93030e958ca43f8420e4f2578c081ae8cb55ec31
Nov 01 10:25:43.075972 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:43.075946    4314 generic.go:356] "Generic (PLEG): container finished" podID="500d094f-6f86-46ec-8869-8372d772e308" containerID="1cf33624bcf2e7d062a58fd18bd8b1e47564b271041b43afbb81d6f680494cee" exitCode=0
Nov 01 10:25:43.076298 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:43.076015    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n" event={"ID":"500d094f-6f86-46ec-8869-8372d772e308","Type":"ContainerDied","Data":"1cf33624bcf2e7d062a58fd18bd8b1e47564b271041b43afbb81d6f680494cee"}
Nov 01 10:25:43.076298 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:43.076052    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n" event={"ID":"500d094f-6f86-46ec-8869-8372d772e308","Type":"ContainerStarted","Data":"34e254169529e5d3f1b7ce8c93030e958ca43f8420e4f2578c081ae8cb55ec31"}
Nov 01 10:25:44.209824 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.209803    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:44.308355 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.308327    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s\" (UniqueName: \"kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s\") pod \"500d094f-6f86-46ec-8869-8372d772e308\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") "
Nov 01 10:25:44.308596 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.308417    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca\") pod \"500d094f-6f86-46ec-8869-8372d772e308\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") "
Nov 01 10:25:44.308596 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.308478    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca\") pod \"500d094f-6f86-46ec-8869-8372d772e308\" (UID: \"500d094f-6f86-46ec-8869-8372d772e308\") "
Nov 01 10:25:44.308887 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.308860    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca" pod "500d094f-6f86-46ec-8869-8372d772e308" (UID: "500d094f-6f86-46ec-8869-8372d772e308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:25:44.310576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.310522    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca" pod "500d094f-6f86-46ec-8869-8372d772e308" (UID: "500d094f-6f86-46ec-8869-8372d772e308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:25:44.310576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.310559    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s" pod "500d094f-6f86-46ec-8869-8372d772e308" (UID: "500d094f-6f86-46ec-8869-8372d772e308"). InnerVolumeSpecName "kube-api-access-6xq6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:25:44.409483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.409418    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xq6s\" (UniqueName: \"kubernetes.io/projected/500d094f-6f86-46ec-8869-8372d772e308-kube-api-access-6xq6s\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:44.409483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.409444    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500d094f-6f86-46ec-8869-8372d772e308-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:44.409483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:44.409454    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500d094f-6f86-46ec-8869-8372d772e308-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:45.088111 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:45.088084    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"
Nov 01 10:25:45.088111 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:45.088105    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n" event={"ID":"500d094f-6f86-46ec-8869-8372d772e308","Type":"ContainerDied","Data":"34e254169529e5d3f1b7ce8c93030e958ca43f8420e4f2578c081ae8cb55ec31"}
Nov 01 10:25:45.088350 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:25:45.088154    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e254169529e5d3f1b7ce8c93030e958ca43f8420e4f2578c081ae8cb55ec31"
Nov 01 10:26:04.793558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:04.792677    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:26:04.793558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:04.792756    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:26:11.321737 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:26:11.321702    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:26:18.309928 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.309891    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"]
Nov 01 10:26:18.310424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.310385    4314 cpu_manager.go:414] "containerMap: removing container" podUID="500d094f-6f86-46ec-8869-8372d772e308" containerName="revoke-keys"
Nov 01 10:26:18.310424 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.310400    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500d094f-6f86-46ec-8869-8372d772e308" containerName="revoke-keys"
Nov 01 10:26:18.310582 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.310481    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="500d094f-6f86-46ec-8869-8372d772e308" containerName="revoke-keys"
Nov 01 10:26:18.313987 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.313960    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.371558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.371429    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.372593 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.372521    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.372771 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.372724    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrfg\" (UniqueName: \"kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.410670 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.410634    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:26:18.410670 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.410634    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:26:18.410937 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.410703    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:26:18.410937 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.410707    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:26:18.413071 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.413046    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"]
Nov 01 10:26:18.474202 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.474172    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrfg\" (UniqueName: \"kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.474394 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.474269    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.474394 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.474316    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.475007 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.474982    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.476885 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.476856    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.531441 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.531416    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrfg\" (UniqueName: \"kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg\") pod \"maas-api-revoke-sub-c6b61a82-7vxwt\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.627702 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.627620    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:18.995728 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:18.995700    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"]
Nov 01 10:26:18.997838 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:26:18.997810    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a2cc7f_015f_4b53_8b50_b17283bbba9b.slice/crio-37f539830355ac0966e2c45a9555a7668f5887962e72f36e2d537fcd7dee1c42 WatchSource:0}: Error finding container 37f539830355ac0966e2c45a9555a7668f5887962e72f36e2d537fcd7dee1c42: Status 404 returned error can't find the container with id 37f539830355ac0966e2c45a9555a7668f5887962e72f36e2d537fcd7dee1c42
Nov 01 10:26:19.280363 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:19.280292    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt" event={"ID":"24a2cc7f-015f-4b53-8b50-b17283bbba9b","Type":"ContainerStarted","Data":"19c54f9485e074eebebd674dd2408f146a59a5b09ceed7de7f5c3f3da7294b4e"}
Nov 01 10:26:19.280480 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:19.280378    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt" event={"ID":"24a2cc7f-015f-4b53-8b50-b17283bbba9b","Type":"ContainerStarted","Data":"37f539830355ac0966e2c45a9555a7668f5887962e72f36e2d537fcd7dee1c42"}
Nov 01 10:26:19.390469 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:19.390413    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt" podStartSLOduration=1.390399934 podStartE2EDuration="1.390399934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:26:19.389327912 +0000 UTC m=+6852.583722468" watchObservedRunningTime="2026-11-01 10:26:19.390399934 +0000 UTC m=+6852.584794509"
Nov 01 10:26:20.285648 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:20.285613    4314 generic.go:356] "Generic (PLEG): container finished" podID="24a2cc7f-015f-4b53-8b50-b17283bbba9b" containerID="19c54f9485e074eebebd674dd2408f146a59a5b09ceed7de7f5c3f3da7294b4e" exitCode=0
Nov 01 10:26:20.285835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:20.285703    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt" event={"ID":"24a2cc7f-015f-4b53-8b50-b17283bbba9b","Type":"ContainerDied","Data":"19c54f9485e074eebebd674dd2408f146a59a5b09ceed7de7f5c3f3da7294b4e"}
Nov 01 10:26:21.417986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.417963    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:21.502167 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.502135    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca\") pod \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") "
Nov 01 10:26:21.502307 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.502268    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca\") pod \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") "
Nov 01 10:26:21.502400 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.502319    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg\" (UniqueName: \"kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg\") pod \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\" (UID: \"24a2cc7f-015f-4b53-8b50-b17283bbba9b\") "
Nov 01 10:26:21.502524 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.502499    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca" pod "24a2cc7f-015f-4b53-8b50-b17283bbba9b" (UID: "24a2cc7f-015f-4b53-8b50-b17283bbba9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:26:21.502667 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.502647    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:26:21.504315 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.504279    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca" pod "24a2cc7f-015f-4b53-8b50-b17283bbba9b" (UID: "24a2cc7f-015f-4b53-8b50-b17283bbba9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:26:21.504452 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.504336    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg" pod "24a2cc7f-015f-4b53-8b50-b17283bbba9b" (UID: "24a2cc7f-015f-4b53-8b50-b17283bbba9b"). InnerVolumeSpecName "kube-api-access-msrfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:26:21.603846 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.603780    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a2cc7f-015f-4b53-8b50-b17283bbba9b-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:26:21.603846 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:21.603806    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrfg\" (UniqueName: \"kubernetes.io/projected/24a2cc7f-015f-4b53-8b50-b17283bbba9b-kube-api-access-msrfg\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:26:22.297132 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:22.297101    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"
Nov 01 10:26:22.297317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:22.297108    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt" event={"ID":"24a2cc7f-015f-4b53-8b50-b17283bbba9b","Type":"ContainerDied","Data":"37f539830355ac0966e2c45a9555a7668f5887962e72f36e2d537fcd7dee1c42"}
Nov 01 10:26:22.297317 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:22.297215    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f539830355ac0966e2c45a9555a7668f5887962e72f36e2d537fcd7dee1c42"
Nov 01 10:26:39.507151 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:39.507113    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:26:41.363828 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:26:41.363776    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:26:47.181794 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:47.181738    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:26:47.182842 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:47.182574    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:26:47.457754 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:47.457657    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:26:47.458033 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:26:47.458002    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:27:02.920354 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.920250    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"]
Nov 01 10:27:02.920797 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.920676    4314 cpu_manager.go:414] "containerMap: removing container" podUID="24a2cc7f-015f-4b53-8b50-b17283bbba9b" containerName="revoke-keys"
Nov 01 10:27:02.920797 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.920687    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a2cc7f-015f-4b53-8b50-b17283bbba9b" containerName="revoke-keys"
Nov 01 10:27:02.920797 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.920759    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a2cc7f-015f-4b53-8b50-b17283bbba9b" containerName="revoke-keys"
Nov 01 10:27:02.924435 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.924411    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:02.973690 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.973664    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:27:02.973849 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.973693    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:27:02.973849 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:02.973720    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:27:03.012745 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.012714    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.012926 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.012798    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.013006 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.012961    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhps\" (UniqueName: \"kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.019827 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.019806    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:27:03.022351 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.022331    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"]
Nov 01 10:27:03.054252 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.054229    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"]
Nov 01 10:27:03.057024 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.057006    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10603c16-q99cx"]
Nov 01 10:27:03.113890 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.113861    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.114023 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.113903    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.114023 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.113991    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhps\" (UniqueName: \"kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.114555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.114509    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.116172 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.116152    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.171666 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.171599    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhps\" (UniqueName: \"kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps\") pod \"maas-api-revoke-sub-d42f98b8-gxc29\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.237878 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.237853    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:03.602856 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:03.602830    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"]
Nov 01 10:27:03.605041 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:27:03.605012    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa5ce07_d1e0_447e_a289_ad761d029a40.slice/crio-75ea9b4d056d5b02f19fc98de6a78024bea179824d0ec8d35c9c7ac1de7ebf16 WatchSource:0}: Error finding container 75ea9b4d056d5b02f19fc98de6a78024bea179824d0ec8d35c9c7ac1de7ebf16: Status 404 returned error can't find the container with id 75ea9b4d056d5b02f19fc98de6a78024bea179824d0ec8d35c9c7ac1de7ebf16
Nov 01 10:27:04.597135 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:04.597050    4314 generic.go:356] "Generic (PLEG): container finished" podID="eaa5ce07-d1e0-447e-a289-ad761d029a40" containerID="8f033374e8822a296292e4c47074496dd79df51d64b306b93695ca9a61d4a88e" exitCode=0
Nov 01 10:27:04.597613 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:04.597122    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29" event={"ID":"eaa5ce07-d1e0-447e-a289-ad761d029a40","Type":"ContainerDied","Data":"8f033374e8822a296292e4c47074496dd79df51d64b306b93695ca9a61d4a88e"}
Nov 01 10:27:04.597613 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:04.597189    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29" event={"ID":"eaa5ce07-d1e0-447e-a289-ad761d029a40","Type":"ContainerStarted","Data":"75ea9b4d056d5b02f19fc98de6a78024bea179824d0ec8d35c9c7ac1de7ebf16"}
Nov 01 10:27:04.927311 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:04.927226    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b429c34-418d-45c1-8714-dbcddd571243" path="/var/lib/kubelet/pods/5b429c34-418d-45c1-8714-dbcddd571243/volumes"
Nov 01 10:27:05.728383 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.728357    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:05.838393 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.838359    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca\") pod \"eaa5ce07-d1e0-447e-a289-ad761d029a40\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") "
Nov 01 10:27:05.838573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.838420    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps\" (UniqueName: \"kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps\") pod \"eaa5ce07-d1e0-447e-a289-ad761d029a40\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") "
Nov 01 10:27:05.838573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.838489    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca\") pod \"eaa5ce07-d1e0-447e-a289-ad761d029a40\" (UID: \"eaa5ce07-d1e0-447e-a289-ad761d029a40\") "
Nov 01 10:27:05.842577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.841383    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps" pod "eaa5ce07-d1e0-447e-a289-ad761d029a40" (UID: "eaa5ce07-d1e0-447e-a289-ad761d029a40"). InnerVolumeSpecName "kube-api-access-tdhps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:27:05.845729 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.845700    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca" pod "eaa5ce07-d1e0-447e-a289-ad761d029a40" (UID: "eaa5ce07-d1e0-447e-a289-ad761d029a40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:27:05.850811 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.850786    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:27:05.851058 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.850941    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdhps\" (UniqueName: \"kubernetes.io/projected/eaa5ce07-d1e0-447e-a289-ad761d029a40-kube-api-access-tdhps\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:27:05.852996 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.852940    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca" pod "eaa5ce07-d1e0-447e-a289-ad761d029a40" (UID: "eaa5ce07-d1e0-447e-a289-ad761d029a40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:27:05.951671 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:05.951635    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5ce07-d1e0-447e-a289-ad761d029a40-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:27:06.607929 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:06.607897    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"
Nov 01 10:27:06.608123 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:06.607934    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29" event={"ID":"eaa5ce07-d1e0-447e-a289-ad761d029a40","Type":"ContainerDied","Data":"75ea9b4d056d5b02f19fc98de6a78024bea179824d0ec8d35c9c7ac1de7ebf16"}
Nov 01 10:27:06.608123 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:06.607989    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ea9b4d056d5b02f19fc98de6a78024bea179824d0ec8d35c9c7ac1de7ebf16"
Nov 01 10:27:11.322394 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:27:11.322355    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:27:13.680632 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.680593    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"]
Nov 01 10:27:13.681114 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.681033    4314 cpu_manager.go:414] "containerMap: removing container" podUID="eaa5ce07-d1e0-447e-a289-ad761d029a40" containerName="revoke-keys"
Nov 01 10:27:13.681114 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.681050    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa5ce07-d1e0-447e-a289-ad761d029a40" containerName="revoke-keys"
Nov 01 10:27:13.681311 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.681185    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa5ce07-d1e0-447e-a289-ad761d029a40" containerName="revoke-keys"
Nov 01 10:27:13.684083 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.684064    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.728338 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.728304    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.728338 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.728343    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.728640 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.728505    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwv5k\" (UniqueName: \"kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.733262 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.733237    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:27:13.779083 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.779051    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:27:13.779259 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.779052    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:27:13.779259 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.779052    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:27:13.781453 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.781433    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"]
Nov 01 10:27:13.829649 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.829622    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.829828 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.829656    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.829828 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.829716    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv5k\" (UniqueName: \"kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.830371 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.830347    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.831959 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.831934    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:13.887577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:13.887522    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv5k\" (UniqueName: \"kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k\") pod \"maas-api-revoke-sub-ee2f6cad-9f4rf\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:14.000841 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:14.000810    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:14.369403 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:14.369375    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"]
Nov 01 10:27:14.371159 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:27:14.371133    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ecbe33_82b5_4948_90dd_5c56b20ee755.slice/crio-8a71c905ddaacd9fb8cbfee87fdb9b95bb0968cf8ddd17e9a483180ef140684a WatchSource:0}: Error finding container 8a71c905ddaacd9fb8cbfee87fdb9b95bb0968cf8ddd17e9a483180ef140684a: Status 404 returned error can't find the container with id 8a71c905ddaacd9fb8cbfee87fdb9b95bb0968cf8ddd17e9a483180ef140684a
Nov 01 10:27:14.648815 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:14.648734    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf" event={"ID":"b8ecbe33-82b5-4948-90dd-5c56b20ee755","Type":"ContainerStarted","Data":"8a71c905ddaacd9fb8cbfee87fdb9b95bb0968cf8ddd17e9a483180ef140684a"}
Nov 01 10:27:15.656203 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:15.656164    4314 generic.go:356] "Generic (PLEG): container finished" podID="b8ecbe33-82b5-4948-90dd-5c56b20ee755" containerID="523b76e1a08566318cfd28e48d11d77824dd0f14c3a98abe3c578b15d13eff84" exitCode=0
Nov 01 10:27:15.656721 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:15.656245    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf" event={"ID":"b8ecbe33-82b5-4948-90dd-5c56b20ee755","Type":"ContainerDied","Data":"523b76e1a08566318cfd28e48d11d77824dd0f14c3a98abe3c578b15d13eff84"}
Nov 01 10:27:16.792525 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.792497    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:16.858849 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.858805    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca\") pod \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") "
Nov 01 10:27:16.859033 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.858909    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca\") pod \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") "
Nov 01 10:27:16.859033 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.858997    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k\" (UniqueName: \"kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k\") pod \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\" (UID: \"b8ecbe33-82b5-4948-90dd-5c56b20ee755\") "
Nov 01 10:27:16.859355 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.859327    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca" pod "b8ecbe33-82b5-4948-90dd-5c56b20ee755" (UID: "b8ecbe33-82b5-4948-90dd-5c56b20ee755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:27:16.861124 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.861097    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca" pod "b8ecbe33-82b5-4948-90dd-5c56b20ee755" (UID: "b8ecbe33-82b5-4948-90dd-5c56b20ee755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:27:16.861251 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.861230    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k" pod "b8ecbe33-82b5-4948-90dd-5c56b20ee755" (UID: "b8ecbe33-82b5-4948-90dd-5c56b20ee755"). InnerVolumeSpecName "kube-api-access-rwv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:27:16.960868 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.960835    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:27:16.960868 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.960865    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ecbe33-82b5-4948-90dd-5c56b20ee755-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:27:16.960868 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:16.960877    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwv5k\" (UniqueName: \"kubernetes.io/projected/b8ecbe33-82b5-4948-90dd-5c56b20ee755-kube-api-access-rwv5k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:27:17.668357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:17.668303    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf" event={"ID":"b8ecbe33-82b5-4948-90dd-5c56b20ee755","Type":"ContainerDied","Data":"8a71c905ddaacd9fb8cbfee87fdb9b95bb0968cf8ddd17e9a483180ef140684a"}
Nov 01 10:27:17.668357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:17.668306    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"
Nov 01 10:27:17.668616 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:17.668373    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a71c905ddaacd9fb8cbfee87fdb9b95bb0968cf8ddd17e9a483180ef140684a"
Nov 01 10:27:27.636103 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:27.636014    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:27:27.791016 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:27.790870    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:27:34.811556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:34.809018    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:27:34.811556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:34.809100    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:27:39.881385 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:39.881353    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:27:41.474784 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:27:41.474709    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:27:59.757247 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.757214    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"]
Nov 01 10:27:59.757750 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.757630    4314 cpu_manager.go:414] "containerMap: removing container" podUID="b8ecbe33-82b5-4948-90dd-5c56b20ee755" containerName="revoke-keys"
Nov 01 10:27:59.757750 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.757642    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ecbe33-82b5-4948-90dd-5c56b20ee755" containerName="revoke-keys"
Nov 01 10:27:59.757750 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.757721    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ecbe33-82b5-4948-90dd-5c56b20ee755" containerName="revoke-keys"
Nov 01 10:27:59.760867 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.760849    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.819691 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.819664    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.819862 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.819712    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tdr\" (UniqueName: \"kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.819862 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.819808    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.858209 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.858187    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:27:59.858352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.858228    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:27:59.858352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.858252    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:27:59.858352 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.858266    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:27:59.860653 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.860637    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"]
Nov 01 10:27:59.920454 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.920434    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.920599 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.920486    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.920663 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.920595    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tdr\" (UniqueName: \"kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.921107 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.921086    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.922784 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.922763    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:27:59.977889 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:27:59.977864    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tdr\" (UniqueName: \"kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr\") pod \"maas-api-revoke-sub-21da78ef-fft7c\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:28:00.073348 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:00.073282    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:28:00.438911 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:00.438889    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"]
Nov 01 10:28:00.440973 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:28:00.440944    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c5f0dc_ec2d_4e12_875d_189df8f0e9b0.slice/crio-7151c58768578c645bc0130f0de38e8261711e6b736eedfde88a7a424f51ae5e WatchSource:0}: Error finding container 7151c58768578c645bc0130f0de38e8261711e6b736eedfde88a7a424f51ae5e: Status 404 returned error can't find the container with id 7151c58768578c645bc0130f0de38e8261711e6b736eedfde88a7a424f51ae5e
Nov 01 10:28:01.045346 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:01.045312    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c" event={"ID":"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0","Type":"ContainerStarted","Data":"23ce368443dfe45a7754060cd62ed782bea0394f54ca27b8012015b3a94440c7"}
Nov 01 10:28:01.045779 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:01.045367    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c" event={"ID":"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0","Type":"ContainerStarted","Data":"7151c58768578c645bc0130f0de38e8261711e6b736eedfde88a7a424f51ae5e"}
Nov 01 10:28:01.154911 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:01.154862    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c" podStartSLOduration=2.1548504250000002 podStartE2EDuration="2.154850425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:28:01.153419124 +0000 UTC m=+6954.347813682" watchObservedRunningTime="2026-11-01 10:28:01.154850425 +0000 UTC m=+6954.349244978"
Nov 01 10:28:01.302926 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:28:01.302861    4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c5f0dc_ec2d_4e12_875d_189df8f0e9b0.slice/crio-conmon-23ce368443dfe45a7754060cd62ed782bea0394f54ca27b8012015b3a94440c7.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 10:28:02.051508 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:02.051477    4314 generic.go:356] "Generic (PLEG): container finished" podID="17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" containerID="23ce368443dfe45a7754060cd62ed782bea0394f54ca27b8012015b3a94440c7" exitCode=0
Nov 01 10:28:02.051920 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:02.051565    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c" event={"ID":"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0","Type":"ContainerDied","Data":"23ce368443dfe45a7754060cd62ed782bea0394f54ca27b8012015b3a94440c7"}
Nov 01 10:28:03.192248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.192222    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:28:03.254383 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.254356    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca\") pod \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") "
Nov 01 10:28:03.254503 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.254394    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca\") pod \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") "
Nov 01 10:28:03.254503 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.254431    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr\" (UniqueName: \"kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr\") pod \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\" (UID: \"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0\") "
Nov 01 10:28:03.254778 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.254752    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca" pod "17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" (UID: "17c5f0dc-ec2d-4e12-875d-189df8f0e9b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:28:03.256547 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.256514    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr" pod "17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" (UID: "17c5f0dc-ec2d-4e12-875d-189df8f0e9b0"). InnerVolumeSpecName "kube-api-access-d6tdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:28:03.256653 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.256572    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca" pod "17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" (UID: "17c5f0dc-ec2d-4e12-875d-189df8f0e9b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:28:03.355577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.355492    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tdr\" (UniqueName: \"kubernetes.io/projected/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-kube-api-access-d6tdr\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:28:03.355577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.355525    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:28:03.355577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:03.355549    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c5f0dc-ec2d-4e12-875d-189df8f0e9b0-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:28:04.062363 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:04.062334    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c"
Nov 01 10:28:04.062568 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:04.062392    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21da78ef-fft7c" event={"ID":"17c5f0dc-ec2d-4e12-875d-189df8f0e9b0","Type":"ContainerDied","Data":"7151c58768578c645bc0130f0de38e8261711e6b736eedfde88a7a424f51ae5e"}
Nov 01 10:28:04.062568 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:04.062440    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7151c58768578c645bc0130f0de38e8261711e6b736eedfde88a7a424f51ae5e"
Nov 01 10:28:10.484988 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.484948    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"]
Nov 01 10:28:10.485577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.485328    4314 cpu_manager.go:414] "containerMap: removing container" podUID="17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" containerName="revoke-keys"
Nov 01 10:28:10.485577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.485339    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" containerName="revoke-keys"
Nov 01 10:28:10.485577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.485425    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c5f0dc-ec2d-4e12-875d-189df8f0e9b0" containerName="revoke-keys"
Nov 01 10:28:10.488776 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.488749    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.539768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.539726    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:28:10.543262 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.543231    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"]
Nov 01 10:28:10.585853 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.585826    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:28:10.585853 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.585831    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:28:10.586085 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.585923    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:28:10.618969 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.618945    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.619085 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.618977    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.619085 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.619007    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnw7r\" (UniqueName: \"kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.719683 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.719653    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnw7r\" (UniqueName: \"kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.719828 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.719762    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.719828 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.719783    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.720810 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.720781    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.729981 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.729956    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.777374 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.777310    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnw7r\" (UniqueName: \"kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r\") pod \"maas-api-revoke-sub-ad11aab6-69llm\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:10.804242 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:10.804220    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:11.172737 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:11.172709    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"]
Nov 01 10:28:11.174676 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:28:11.174647    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0047fc77_ce14_4867_821b_aca7ae3254a9.slice/crio-9f60e465a4a0806b0852422d07cb2d53149d68f29c2176a35ebcb00657964615 WatchSource:0}: Error finding container 9f60e465a4a0806b0852422d07cb2d53149d68f29c2176a35ebcb00657964615: Status 404 returned error can't find the container with id 9f60e465a4a0806b0852422d07cb2d53149d68f29c2176a35ebcb00657964615
Nov 01 10:28:11.323576 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:28:11.323522    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:28:11.911429 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:11.911399    4314 scope.go:122] "RemoveContainer" containerID="3387f939c5427b567be89ed73de6ed5e26f7368ce705756325a8c1f05208e0f8"
Nov 01 10:28:12.103670 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:12.103639    4314 generic.go:356] "Generic (PLEG): container finished" podID="0047fc77-ce14-4867-821b-aca7ae3254a9" containerID="45f4c79adf975274ca01bd6f0d7d6fcca42c584fc75968226df4badf904b555f" exitCode=0
Nov 01 10:28:12.103833 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:12.103691    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm" event={"ID":"0047fc77-ce14-4867-821b-aca7ae3254a9","Type":"ContainerDied","Data":"45f4c79adf975274ca01bd6f0d7d6fcca42c584fc75968226df4badf904b555f"}
Nov 01 10:28:12.103833 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:12.103733    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm" event={"ID":"0047fc77-ce14-4867-821b-aca7ae3254a9","Type":"ContainerStarted","Data":"9f60e465a4a0806b0852422d07cb2d53149d68f29c2176a35ebcb00657964615"}
Nov 01 10:28:13.317069 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.317030    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:13.448028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.447188    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r\" (UniqueName: \"kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r\") pod \"0047fc77-ce14-4867-821b-aca7ae3254a9\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") "
Nov 01 10:28:13.448028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.447262    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca\") pod \"0047fc77-ce14-4867-821b-aca7ae3254a9\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") "
Nov 01 10:28:13.448028 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.447307    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca\") pod \"0047fc77-ce14-4867-821b-aca7ae3254a9\" (UID: \"0047fc77-ce14-4867-821b-aca7ae3254a9\") "
Nov 01 10:28:13.448524 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.448476    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca" pod "0047fc77-ce14-4867-821b-aca7ae3254a9" (UID: "0047fc77-ce14-4867-821b-aca7ae3254a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:28:13.450738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.450698    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca" pod "0047fc77-ce14-4867-821b-aca7ae3254a9" (UID: "0047fc77-ce14-4867-821b-aca7ae3254a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:28:13.450871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.450787    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r" pod "0047fc77-ce14-4867-821b-aca7ae3254a9" (UID: "0047fc77-ce14-4867-821b-aca7ae3254a9"). InnerVolumeSpecName "kube-api-access-cnw7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:28:13.548833 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.548739    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:28:13.548833 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.548778    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0047fc77-ce14-4867-821b-aca7ae3254a9-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:28:13.548833 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:13.548799    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnw7r\" (UniqueName: \"kubernetes.io/projected/0047fc77-ce14-4867-821b-aca7ae3254a9-kube-api-access-cnw7r\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:28:14.131387 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:14.131342    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm" event={"ID":"0047fc77-ce14-4867-821b-aca7ae3254a9","Type":"ContainerDied","Data":"9f60e465a4a0806b0852422d07cb2d53149d68f29c2176a35ebcb00657964615"}
Nov 01 10:28:14.131603 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:14.131409    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f60e465a4a0806b0852422d07cb2d53149d68f29c2176a35ebcb00657964615"
Nov 01 10:28:14.131603 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:14.131482    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad11aab6-69llm"
Nov 01 10:28:32.185731 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:32.185637    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:28:32.186600 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:32.186564    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:28:32.448730 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:32.448639    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:28:32.448730 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:32.448710    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:28:40.252409 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:40.252374    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:28:40.429719 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:40.429688    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:28:41.322220 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:28:41.322184    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:28:49.802559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:49.799627    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:28:49.802559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:28:49.799696    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:29:01.859380 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.859346    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"]
Nov 01 10:29:01.859922 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.859751    4314 cpu_manager.go:414] "containerMap: removing container" podUID="0047fc77-ce14-4867-821b-aca7ae3254a9" containerName="revoke-keys"
Nov 01 10:29:01.859922 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.859764    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0047fc77-ce14-4867-821b-aca7ae3254a9" containerName="revoke-keys"
Nov 01 10:29:01.859922 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.859835    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0047fc77-ce14-4867-821b-aca7ae3254a9" containerName="revoke-keys"
Nov 01 10:29:01.863190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.863169    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:01.918174 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.918141    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"]
Nov 01 10:29:01.960006 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.959983    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:29:01.960189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.960088    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:29:01.960189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.960144    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:29:01.960189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.960142    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:29:01.988775 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.988751    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:01.988921 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.988788    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9sb\" (UniqueName: \"kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:01.988970 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:01.988915    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.089955 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.089929    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.090091 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.089990    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.090091 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.090011    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9sb\" (UniqueName: \"kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.090572 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.090520    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.092274 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.092248    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.147527 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.147473    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9sb\" (UniqueName: \"kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb\") pod \"maas-api-revoke-sub-6d14c8bb-zzhcg\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.176482 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.176459    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:02.538882 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:02.538856    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"]
Nov 01 10:29:02.540709 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:29:02.540673    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccf1125_1922_4898_94f9_6dbfe0b3b1d8.slice/crio-b2b0e1e3d575a1e84d98bc2a49f66bb8f68fbaffc626f38f25b63b56938e9186 WatchSource:0}: Error finding container b2b0e1e3d575a1e84d98bc2a49f66bb8f68fbaffc626f38f25b63b56938e9186: Status 404 returned error can't find the container with id b2b0e1e3d575a1e84d98bc2a49f66bb8f68fbaffc626f38f25b63b56938e9186
Nov 01 10:29:03.551651 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:03.551619    4314 generic.go:356] "Generic (PLEG): container finished" podID="4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" containerID="7e4c96366f4190d0268e42e8973a4efb23122477496cfc5c8ba8aed0174a5ec0" exitCode=0
Nov 01 10:29:03.552034 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:03.551673    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg" event={"ID":"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8","Type":"ContainerDied","Data":"7e4c96366f4190d0268e42e8973a4efb23122477496cfc5c8ba8aed0174a5ec0"}
Nov 01 10:29:03.552034 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:03.551723    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg" event={"ID":"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8","Type":"ContainerStarted","Data":"b2b0e1e3d575a1e84d98bc2a49f66bb8f68fbaffc626f38f25b63b56938e9186"}
Nov 01 10:29:04.689752 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.689725    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:04.816456 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.816385    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca\") pod \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") "
Nov 01 10:29:04.816456 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.816439    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb\" (UniqueName: \"kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb\") pod \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") "
Nov 01 10:29:04.816687 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.816497    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca\") pod \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\" (UID: \"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8\") "
Nov 01 10:29:04.816943 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.816914    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca" pod "4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" (UID: "4ccf1125-1922-4898-94f9-6dbfe0b3b1d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:29:04.818503 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.818475    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca" pod "4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" (UID: "4ccf1125-1922-4898-94f9-6dbfe0b3b1d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:29:04.818503 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.818489    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb" pod "4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" (UID: "4ccf1125-1922-4898-94f9-6dbfe0b3b1d8"). InnerVolumeSpecName "kube-api-access-fg9sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:29:04.918070 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.918041    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:04.918070 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.918067    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:04.918070 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:04.918076    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9sb\" (UniqueName: \"kubernetes.io/projected/4ccf1125-1922-4898-94f9-6dbfe0b3b1d8-kube-api-access-fg9sb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:05.564023 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:05.563990    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg" event={"ID":"4ccf1125-1922-4898-94f9-6dbfe0b3b1d8","Type":"ContainerDied","Data":"b2b0e1e3d575a1e84d98bc2a49f66bb8f68fbaffc626f38f25b63b56938e9186"}
Nov 01 10:29:05.564023 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:05.564008    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d14c8bb-zzhcg"
Nov 01 10:29:05.564260 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:05.564039    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2b0e1e3d575a1e84d98bc2a49f66bb8f68fbaffc626f38f25b63b56938e9186"
Nov 01 10:29:11.324380 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:29:11.324328    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:29:12.967008 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:12.966978    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"]
Nov 01 10:29:12.967581 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:12.967561    4314 cpu_manager.go:414] "containerMap: removing container" podUID="4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" containerName="revoke-keys"
Nov 01 10:29:12.967705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:12.967581    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" containerName="revoke-keys"
Nov 01 10:29:12.967705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:12.967678    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccf1125-1922-4898-94f9-6dbfe0b3b1d8" containerName="revoke-keys"
Nov 01 10:29:12.970919 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:12.970890    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.024635 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.024602    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"]
Nov 01 10:29:13.067361 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.067317    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:29:13.067361 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.067347    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:29:13.067361 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.067317    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:29:13.067632 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.067402    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:29:13.099095 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.099066    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm28k\" (UniqueName: \"kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.099234 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.099144    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.099279 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.099244    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.200793 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.200755    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm28k\" (UniqueName: \"kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.200982 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.200851    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.200982 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.200946    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.201492 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.201471    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.203304 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.203283    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.258108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.258081    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm28k\" (UniqueName: \"kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k\") pod \"maas-api-revoke-sub-47c1d6cd-jrswv\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.283916 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.283891    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:13.653757 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:13.653655    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"]
Nov 01 10:29:14.625816 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:14.625777    4314 generic.go:356] "Generic (PLEG): container finished" podID="40f2ea1c-d56c-40f5-b34d-171c7210fcd4" containerID="8f06bc07caaf8d416bd6187d4aeff59415170e179ba993c6245a0b22f6055f86" exitCode=0
Nov 01 10:29:14.626259 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:14.625821    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv" event={"ID":"40f2ea1c-d56c-40f5-b34d-171c7210fcd4","Type":"ContainerDied","Data":"8f06bc07caaf8d416bd6187d4aeff59415170e179ba993c6245a0b22f6055f86"}
Nov 01 10:29:14.626259 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:14.625904    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv" event={"ID":"40f2ea1c-d56c-40f5-b34d-171c7210fcd4","Type":"ContainerStarted","Data":"c136370911ef1da763e179e07e99e7687d80d6e08203ed645e49f14e11aa6d27"}
Nov 01 10:29:15.761454 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.761430    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:15.927483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.927397    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k\" (UniqueName: \"kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k\") pod \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") "
Nov 01 10:29:15.927483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.927460    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca\") pod \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") "
Nov 01 10:29:15.927747 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.927601    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca\") pod \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\" (UID: \"40f2ea1c-d56c-40f5-b34d-171c7210fcd4\") "
Nov 01 10:29:15.927938 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.927913    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca" pod "40f2ea1c-d56c-40f5-b34d-171c7210fcd4" (UID: "40f2ea1c-d56c-40f5-b34d-171c7210fcd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:29:15.929558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.929512    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca" pod "40f2ea1c-d56c-40f5-b34d-171c7210fcd4" (UID: "40f2ea1c-d56c-40f5-b34d-171c7210fcd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:29:15.929665 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:15.929593    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k" pod "40f2ea1c-d56c-40f5-b34d-171c7210fcd4" (UID: "40f2ea1c-d56c-40f5-b34d-171c7210fcd4"). InnerVolumeSpecName "kube-api-access-wm28k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:29:16.028619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:16.028594    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:16.028619 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:16.028617    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:16.028764 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:16.028626    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm28k\" (UniqueName: \"kubernetes.io/projected/40f2ea1c-d56c-40f5-b34d-171c7210fcd4-kube-api-access-wm28k\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:16.638486 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:16.638451    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv" event={"ID":"40f2ea1c-d56c-40f5-b34d-171c7210fcd4","Type":"ContainerDied","Data":"c136370911ef1da763e179e07e99e7687d80d6e08203ed645e49f14e11aa6d27"}
Nov 01 10:29:16.638486 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:16.638481    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c1d6cd-jrswv"
Nov 01 10:29:16.638726 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:16.638502    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c136370911ef1da763e179e07e99e7687d80d6e08203ed645e49f14e11aa6d27"
Nov 01 10:29:19.796680 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:19.796637    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:29:19.797304 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:19.797271    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:29:23.639248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.639210    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"]
Nov 01 10:29:23.639649 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.639614    4314 cpu_manager.go:414] "containerMap: removing container" podUID="40f2ea1c-d56c-40f5-b34d-171c7210fcd4" containerName="revoke-keys"
Nov 01 10:29:23.639649 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.639626    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f2ea1c-d56c-40f5-b34d-171c7210fcd4" containerName="revoke-keys"
Nov 01 10:29:23.639726 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.639688    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f2ea1c-d56c-40f5-b34d-171c7210fcd4" containerName="revoke-keys"
Nov 01 10:29:23.643047 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.643024    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.694054 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.694022    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:29:23.694054 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.694027    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:29:23.739883 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.739853    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:29:23.739883 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.739872    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:29:23.742719 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.742700    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"]
Nov 01 10:29:23.762304 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.762279    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv478\" (UniqueName: \"kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.762477 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.762336    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.762528 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.762477    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.863905 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.863868    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv478\" (UniqueName: \"kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.864064 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.863916    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.864064 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.863952    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.864682 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.864662    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.866305 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.866286    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.921357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.921294    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv478\" (UniqueName: \"kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478\") pod \"maas-api-revoke-sub-61f83c3e-bpj7p\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:23.959951 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:23.959923    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:24.323964 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:24.323941    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"]
Nov 01 10:29:24.326041 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:29:24.326014    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc85bc4b_a2ba_4c86_90f8_6bf795b142c3.slice/crio-a934be77693e896aea39e3ed6553cd64abd54fe0a5b117940257110bb3ff93e0 WatchSource:0}: Error finding container a934be77693e896aea39e3ed6553cd64abd54fe0a5b117940257110bb3ff93e0: Status 404 returned error can't find the container with id a934be77693e896aea39e3ed6553cd64abd54fe0a5b117940257110bb3ff93e0
Nov 01 10:29:24.734804 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:24.734773    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p" event={"ID":"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3","Type":"ContainerStarted","Data":"110b5d8f32e4a480737373bd4600af1b04fe9ae6f567a92e65149330d1cd451a"}
Nov 01 10:29:24.735224 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:24.734823    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p" event={"ID":"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3","Type":"ContainerStarted","Data":"a934be77693e896aea39e3ed6553cd64abd54fe0a5b117940257110bb3ff93e0"}
Nov 01 10:29:24.845895 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:24.845794    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p" podStartSLOduration=1.845779345 podStartE2EDuration="1.845779345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:29:24.843306531 +0000 UTC m=+7038.037701098" watchObservedRunningTime="2026-11-01 10:29:24.845779345 +0000 UTC m=+7038.040173898"
Nov 01 10:29:25.742290 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:25.742259    4314 generic.go:356] "Generic (PLEG): container finished" podID="dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" containerID="110b5d8f32e4a480737373bd4600af1b04fe9ae6f567a92e65149330d1cd451a" exitCode=0
Nov 01 10:29:25.742725 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:25.742323    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p" event={"ID":"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3","Type":"ContainerDied","Data":"110b5d8f32e4a480737373bd4600af1b04fe9ae6f567a92e65149330d1cd451a"}
Nov 01 10:29:26.878834 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.878805    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:26.890158 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.890131    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478\" (UniqueName: \"kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478\") pod \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") "
Nov 01 10:29:26.890299 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.890185    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca\") pod \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") "
Nov 01 10:29:26.890299 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.890289    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca\") pod \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\" (UID: \"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3\") "
Nov 01 10:29:26.890527 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.890501    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca" pod "dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" (UID: "dc85bc4b-a2ba-4c86-90f8-6bf795b142c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:29:26.890648 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.890633    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:26.892098 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.892080    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478" pod "dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" (UID: "dc85bc4b-a2ba-4c86-90f8-6bf795b142c3"). InnerVolumeSpecName "kube-api-access-vv478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:29:26.892247 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.892228    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca" pod "dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" (UID: "dc85bc4b-a2ba-4c86-90f8-6bf795b142c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:29:26.992070 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.992036    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:26.992213 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:26.992073    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv478\" (UniqueName: \"kubernetes.io/projected/dc85bc4b-a2ba-4c86-90f8-6bf795b142c3-kube-api-access-vv478\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:27.754051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:27.754024    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p"
Nov 01 10:29:27.754231 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:27.754019    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f83c3e-bpj7p" event={"ID":"dc85bc4b-a2ba-4c86-90f8-6bf795b142c3","Type":"ContainerDied","Data":"a934be77693e896aea39e3ed6553cd64abd54fe0a5b117940257110bb3ff93e0"}
Nov 01 10:29:27.754231 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:27.754145    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a934be77693e896aea39e3ed6553cd64abd54fe0a5b117940257110bb3ff93e0"
Nov 01 10:29:34.594968 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.594937    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"]
Nov 01 10:29:34.595816 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.595421    4314 cpu_manager.go:414] "containerMap: removing container" podUID="dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" containerName="revoke-keys"
Nov 01 10:29:34.595816 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.595436    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" containerName="revoke-keys"
Nov 01 10:29:34.595816 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.595514    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc85bc4b-a2ba-4c86-90f8-6bf795b142c3" containerName="revoke-keys"
Nov 01 10:29:34.598663 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.598642    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.653423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.653390    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"]
Nov 01 10:29:34.661040 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.661013    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.661326 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.661302    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.661458 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.661441    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqgv\" (UniqueName: \"kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.695936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.695904    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:29:34.695936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.695905    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:29:34.695936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.695905    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:29:34.696162 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.696147    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:29:34.762568 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.762523    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.762693 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.762579    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.762693 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.762602    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqgv\" (UniqueName: \"kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.763282 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.763258    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.764909 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.764888    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.819817 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.819789    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqgv\" (UniqueName: \"kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv\") pod \"maas-api-revoke-sub-e539d9ea-tgvgw\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:34.919608 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:34.919554    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:35.284651 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:35.284625    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"]
Nov 01 10:29:35.286333 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:29:35.286305    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095a6e12_7129_4eee_9842_d74276ec0441.slice/crio-97edb7cf61640bde24501627c62ab2d419bc0a995f887d6c44219c485b8b6bec WatchSource:0}: Error finding container 97edb7cf61640bde24501627c62ab2d419bc0a995f887d6c44219c485b8b6bec: Status 404 returned error can't find the container with id 97edb7cf61640bde24501627c62ab2d419bc0a995f887d6c44219c485b8b6bec
Nov 01 10:29:35.798936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:35.798899    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw" event={"ID":"095a6e12-7129-4eee-9842-d74276ec0441","Type":"ContainerStarted","Data":"c144079c6f5769d23427603a464d9e3bbadfbcdc278a1aba1aff8f129ec6e1b1"}
Nov 01 10:29:35.799330 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:35.798960    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw" event={"ID":"095a6e12-7129-4eee-9842-d74276ec0441","Type":"ContainerStarted","Data":"97edb7cf61640bde24501627c62ab2d419bc0a995f887d6c44219c485b8b6bec"}
Nov 01 10:29:35.909184 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:35.909130    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw" podStartSLOduration=1.909116246 podStartE2EDuration="1.909116246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:29:35.906703017 +0000 UTC m=+7049.101097578" watchObservedRunningTime="2026-11-01 10:29:35.909116246 +0000 UTC m=+7049.103510800"
Nov 01 10:29:36.805739 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:36.805709    4314 generic.go:356] "Generic (PLEG): container finished" podID="095a6e12-7129-4eee-9842-d74276ec0441" containerID="c144079c6f5769d23427603a464d9e3bbadfbcdc278a1aba1aff8f129ec6e1b1" exitCode=0
Nov 01 10:29:36.806140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:36.805785    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw" event={"ID":"095a6e12-7129-4eee-9842-d74276ec0441","Type":"ContainerDied","Data":"c144079c6f5769d23427603a464d9e3bbadfbcdc278a1aba1aff8f129ec6e1b1"}
Nov 01 10:29:38.010133 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.010100    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:38.099610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.098871    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv\" (UniqueName: \"kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv\") pod \"095a6e12-7129-4eee-9842-d74276ec0441\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") "
Nov 01 10:29:38.099610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.099008    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca\") pod \"095a6e12-7129-4eee-9842-d74276ec0441\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") "
Nov 01 10:29:38.099610 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.099076    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca\") pod \"095a6e12-7129-4eee-9842-d74276ec0441\" (UID: \"095a6e12-7129-4eee-9842-d74276ec0441\") "
Nov 01 10:29:38.099946 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.099835    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca" pod "095a6e12-7129-4eee-9842-d74276ec0441" (UID: "095a6e12-7129-4eee-9842-d74276ec0441"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:29:38.129174 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.129125    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv" pod "095a6e12-7129-4eee-9842-d74276ec0441" (UID: "095a6e12-7129-4eee-9842-d74276ec0441"). InnerVolumeSpecName "kube-api-access-vnqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:29:38.135360 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.135322    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca" pod "095a6e12-7129-4eee-9842-d74276ec0441" (UID: "095a6e12-7129-4eee-9842-d74276ec0441"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:29:38.209260 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.209166    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnqgv\" (UniqueName: \"kubernetes.io/projected/095a6e12-7129-4eee-9842-d74276ec0441-kube-api-access-vnqgv\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:38.209260 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.209208    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:38.209260 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.209226    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095a6e12-7129-4eee-9842-d74276ec0441-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:38.846168 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.846131    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw" event={"ID":"095a6e12-7129-4eee-9842-d74276ec0441","Type":"ContainerDied","Data":"97edb7cf61640bde24501627c62ab2d419bc0a995f887d6c44219c485b8b6bec"}
Nov 01 10:29:38.846470 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.846442    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97edb7cf61640bde24501627c62ab2d419bc0a995f887d6c44219c485b8b6bec"
Nov 01 10:29:38.846692 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:38.846679    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e539d9ea-tgvgw"
Nov 01 10:29:39.640078 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:39.640046    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bx5ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:29:39.641234 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:39.640751    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bx5ck" podUID="d68734d8-e154-4730-990c-3bb7292efaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:29:40.610555 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:40.609818    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:29:41.321466 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:29:41.321428    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:29:45.094926 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.094887    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:29:45.553354 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.553317    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"]
Nov 01 10:29:45.553776 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.553762    4314 cpu_manager.go:414] "containerMap: removing container" podUID="095a6e12-7129-4eee-9842-d74276ec0441" containerName="revoke-keys"
Nov 01 10:29:45.553835 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.553777    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095a6e12-7129-4eee-9842-d74276ec0441" containerName="revoke-keys"
Nov 01 10:29:45.553881 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.553854    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="095a6e12-7129-4eee-9842-d74276ec0441" containerName="revoke-keys"
Nov 01 10:29:45.557169 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.557146    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.631797 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.631764    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.631966 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.631812    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.631966 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.631910    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzdb\" (UniqueName: \"kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.653738 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.653710    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:29:45.653885 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.653779    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:29:45.653885 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.653805    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:29:45.653970 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.653886    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:29:45.656573 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.656554    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"]
Nov 01 10:29:45.732873 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.732846    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzdb\" (UniqueName: \"kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.733032 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.732914    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.733032 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.732945    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.733604 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.733577    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.735312 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.735287    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.789840 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.789820    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzdb\" (UniqueName: \"kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb\") pod \"maas-api-revoke-sub-8e5631b3-tlkpw\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:45.869936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:45.869869    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:46.234518 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:46.234486    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"]
Nov 01 10:29:46.239990 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:29:46.236213    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac980a5f_f55d_4452_bd4d_4de7f586237f.slice/crio-517d705a5a0f97ab1bdaaa935f91bfe490a559873868a585755aa9695ecb0a0b WatchSource:0}: Error finding container 517d705a5a0f97ab1bdaaa935f91bfe490a559873868a585755aa9695ecb0a0b: Status 404 returned error can't find the container with id 517d705a5a0f97ab1bdaaa935f91bfe490a559873868a585755aa9695ecb0a0b
Nov 01 10:29:46.893943 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:46.893865    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw" event={"ID":"ac980a5f-f55d-4452-bd4d-4de7f586237f","Type":"ContainerStarted","Data":"f63ac137bcd1679aa8eea6fdb9a8221b6c2e007ab0e976c2a830e2ac6f7d344d"}
Nov 01 10:29:46.893943 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:46.893912    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw" event={"ID":"ac980a5f-f55d-4452-bd4d-4de7f586237f","Type":"ContainerStarted","Data":"517d705a5a0f97ab1bdaaa935f91bfe490a559873868a585755aa9695ecb0a0b"}
Nov 01 10:29:47.003152 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:47.003100    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw" podStartSLOduration=2.003085755 podStartE2EDuration="2.003085755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:29:47.002079037 +0000 UTC m=+7060.196473589" watchObservedRunningTime="2026-11-01 10:29:47.003085755 +0000 UTC m=+7060.197480309"
Nov 01 10:29:47.904550 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:47.904506    4314 generic.go:356] "Generic (PLEG): container finished" podID="ac980a5f-f55d-4452-bd4d-4de7f586237f" containerID="f63ac137bcd1679aa8eea6fdb9a8221b6c2e007ab0e976c2a830e2ac6f7d344d" exitCode=0
Nov 01 10:29:47.905105 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:47.904604    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw" event={"ID":"ac980a5f-f55d-4452-bd4d-4de7f586237f","Type":"ContainerDied","Data":"f63ac137bcd1679aa8eea6fdb9a8221b6c2e007ab0e976c2a830e2ac6f7d344d"}
Nov 01 10:29:49.300043 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.299729    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:49.388685 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.387806    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb\" (UniqueName: \"kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb\") pod \"ac980a5f-f55d-4452-bd4d-4de7f586237f\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") "
Nov 01 10:29:49.388685 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.387930    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca\") pod \"ac980a5f-f55d-4452-bd4d-4de7f586237f\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") "
Nov 01 10:29:49.388685 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.387966    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca\") pod \"ac980a5f-f55d-4452-bd4d-4de7f586237f\" (UID: \"ac980a5f-f55d-4452-bd4d-4de7f586237f\") "
Nov 01 10:29:49.393779 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.393753    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca" pod "ac980a5f-f55d-4452-bd4d-4de7f586237f" (UID: "ac980a5f-f55d-4452-bd4d-4de7f586237f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:29:49.416235 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.416189    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca" pod "ac980a5f-f55d-4452-bd4d-4de7f586237f" (UID: "ac980a5f-f55d-4452-bd4d-4de7f586237f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:29:49.416579 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.416526    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb" pod "ac980a5f-f55d-4452-bd4d-4de7f586237f" (UID: "ac980a5f-f55d-4452-bd4d-4de7f586237f"). InnerVolumeSpecName "kube-api-access-2wzdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:29:49.491554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.489170    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:49.491554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.489211    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac980a5f-f55d-4452-bd4d-4de7f586237f-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:49.491554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.489229    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzdb\" (UniqueName: \"kubernetes.io/projected/ac980a5f-f55d-4452-bd4d-4de7f586237f-kube-api-access-2wzdb\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:29:49.922038 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.921944    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw" event={"ID":"ac980a5f-f55d-4452-bd4d-4de7f586237f","Type":"ContainerDied","Data":"517d705a5a0f97ab1bdaaa935f91bfe490a559873868a585755aa9695ecb0a0b"}
Nov 01 10:29:49.922335 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.922314    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517d705a5a0f97ab1bdaaa935f91bfe490a559873868a585755aa9695ecb0a0b"
Nov 01 10:29:49.922513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:49.922499    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5631b3-tlkpw"
Nov 01 10:29:56.520198 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.520157    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"]
Nov 01 10:29:56.520843 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.520821    4314 cpu_manager.go:414] "containerMap: removing container" podUID="ac980a5f-f55d-4452-bd4d-4de7f586237f" containerName="revoke-keys"
Nov 01 10:29:56.520949 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.520844    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac980a5f-f55d-4452-bd4d-4de7f586237f" containerName="revoke-keys"
Nov 01 10:29:56.521006 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.520954    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac980a5f-f55d-4452-bd4d-4de7f586237f" containerName="revoke-keys"
Nov 01 10:29:56.524351 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.524328    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.560942 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.560917    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.561047 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.560975    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.561091 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.561050    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xspw\" (UniqueName: \"kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.620705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.620668    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:29:56.620705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.620670    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:29:56.620930 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.620735    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:29:56.620930 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.620831    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:29:56.623572 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.623552    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"]
Nov 01 10:29:56.662176 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.662154    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.662300 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.662215    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.662300 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.662240    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xspw\" (UniqueName: \"kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.662870 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.662851    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.664568 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.664547    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.719768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.719744    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xspw\" (UniqueName: \"kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw\") pod \"maas-api-revoke-sub-a15030fc-4fw6l\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:56.836871 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:56.836794    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:29:57.202410 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:57.202380    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"]
Nov 01 10:29:57.204224 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:29:57.204197    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730b33a3_b3e3_4c83_bd68_977d1cf50500.slice/crio-1646bb19b49354b08c722038f0148ddea11c87ed71429a83145754d179918ad4 WatchSource:0}: Error finding container 1646bb19b49354b08c722038f0148ddea11c87ed71429a83145754d179918ad4: Status 404 returned error can't find the container with id 1646bb19b49354b08c722038f0148ddea11c87ed71429a83145754d179918ad4
Nov 01 10:29:58.004210 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:58.004180    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l" event={"ID":"730b33a3-b3e3-4c83-bd68-977d1cf50500","Type":"ContainerStarted","Data":"ec6eaec2dfb34706bdc37d24b67584b9520a858f84a509f70fb2f8a90f02d7f9"}
Nov 01 10:29:58.004660 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:58.004238    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l" event={"ID":"730b33a3-b3e3-4c83-bd68-977d1cf50500","Type":"ContainerStarted","Data":"1646bb19b49354b08c722038f0148ddea11c87ed71429a83145754d179918ad4"}
Nov 01 10:29:58.113392 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:58.113339    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l" podStartSLOduration=2.113323066 podStartE2EDuration="2.113323066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:29:58.112584609 +0000 UTC m=+7071.306979166" watchObservedRunningTime="2026-11-01 10:29:58.113323066 +0000 UTC m=+7071.307717621"
Nov 01 10:29:59.014669 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:59.014630    4314 generic.go:356] "Generic (PLEG): container finished" podID="730b33a3-b3e3-4c83-bd68-977d1cf50500" containerID="ec6eaec2dfb34706bdc37d24b67584b9520a858f84a509f70fb2f8a90f02d7f9" exitCode=0
Nov 01 10:29:59.015194 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:29:59.014721    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l" event={"ID":"730b33a3-b3e3-4c83-bd68-977d1cf50500","Type":"ContainerDied","Data":"ec6eaec2dfb34706bdc37d24b67584b9520a858f84a509f70fb2f8a90f02d7f9"}
Nov 01 10:30:00.576466 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.576438    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:30:00.702556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.697991    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca\") pod \"730b33a3-b3e3-4c83-bd68-977d1cf50500\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") "
Nov 01 10:30:00.702556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.698071    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw\" (UniqueName: \"kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw\") pod \"730b33a3-b3e3-4c83-bd68-977d1cf50500\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") "
Nov 01 10:30:00.702556 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.698146    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca\") pod \"730b33a3-b3e3-4c83-bd68-977d1cf50500\" (UID: \"730b33a3-b3e3-4c83-bd68-977d1cf50500\") "
Nov 01 10:30:00.722558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.716670    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw" pod "730b33a3-b3e3-4c83-bd68-977d1cf50500" (UID: "730b33a3-b3e3-4c83-bd68-977d1cf50500"). InnerVolumeSpecName "kube-api-access-6xspw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:30:00.739554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.735659    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca" pod "730b33a3-b3e3-4c83-bd68-977d1cf50500" (UID: "730b33a3-b3e3-4c83-bd68-977d1cf50500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:30:00.791601 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.791372    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xspw\" (UniqueName: \"kubernetes.io/projected/730b33a3-b3e3-4c83-bd68-977d1cf50500-kube-api-access-6xspw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:00.791601 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.791467    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:00.791601 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.791437    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca" pod "730b33a3-b3e3-4c83-bd68-977d1cf50500" (UID: "730b33a3-b3e3-4c83-bd68-977d1cf50500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:30:00.922557 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:00.919338    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730b33a3-b3e3-4c83-bd68-977d1cf50500-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:01.060181 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:01.060074    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l" event={"ID":"730b33a3-b3e3-4c83-bd68-977d1cf50500","Type":"ContainerDied","Data":"1646bb19b49354b08c722038f0148ddea11c87ed71429a83145754d179918ad4"}
Nov 01 10:30:01.060487 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:01.060446    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1646bb19b49354b08c722038f0148ddea11c87ed71429a83145754d179918ad4"
Nov 01 10:30:01.060785 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:01.060726    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a15030fc-4fw6l"
Nov 01 10:30:02.185667 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:02.185628    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:30:02.186311 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:02.186278    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:30:02.456743 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:02.456648    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-h8njt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:30:02.457026 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:02.456994    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-h8njt" podUID="3959e93c-0abf-409a-9f46-d31930d382c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:30:07.307683 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.307650    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"]
Nov 01 10:30:07.308148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.308023    4314 cpu_manager.go:414] "containerMap: removing container" podUID="730b33a3-b3e3-4c83-bd68-977d1cf50500" containerName="revoke-keys"
Nov 01 10:30:07.308148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.308034    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730b33a3-b3e3-4c83-bd68-977d1cf50500" containerName="revoke-keys"
Nov 01 10:30:07.308148 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.308110    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="730b33a3-b3e3-4c83-bd68-977d1cf50500" containerName="revoke-keys"
Nov 01 10:30:07.313623 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.313521    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.365070 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.365041    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"]
Nov 01 10:30:07.408041 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.408015    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:30:07.408191 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.408015    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:30:07.408191 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.408103    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:30:07.408621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.408603    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:30:07.414071 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.414051    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspqh\" (UniqueName: \"kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.414155 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.414087    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.414155 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.414108    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.514876 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.514846    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspqh\" (UniqueName: \"kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.515035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.514880    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.515035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.514897    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.515452 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.515432    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.517163 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.517138    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.571913 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.571857    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspqh\" (UniqueName: \"kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh\") pod \"maas-api-revoke-sub-136a1178-49g8c\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.628219 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.628198    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:07.993382 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:07.993353    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"]
Nov 01 10:30:07.994891 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:30:07.994869    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183ae060_2054_42ff_ac8d_c674b2315c37.slice/crio-706dc4dec1af94643661a352e5fa73920fdc5c67ec966fdac3fc07e33f8115b7 WatchSource:0}: Error finding container 706dc4dec1af94643661a352e5fa73920fdc5c67ec966fdac3fc07e33f8115b7: Status 404 returned error can't find the container with id 706dc4dec1af94643661a352e5fa73920fdc5c67ec966fdac3fc07e33f8115b7
Nov 01 10:30:08.132212 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:08.132179    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c" event={"ID":"183ae060-2054-42ff-ac8d-c674b2315c37","Type":"ContainerStarted","Data":"706dc4dec1af94643661a352e5fa73920fdc5c67ec966fdac3fc07e33f8115b7"}
Nov 01 10:30:09.139140 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:09.139063    4314 generic.go:356] "Generic (PLEG): container finished" podID="183ae060-2054-42ff-ac8d-c674b2315c37" containerID="65bc78e05964b108042b3cbd09d7faa4f6177e84914674180d9f2c64e299a7d6" exitCode=0
Nov 01 10:30:09.139506 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:09.139144    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c" event={"ID":"183ae060-2054-42ff-ac8d-c674b2315c37","Type":"ContainerDied","Data":"65bc78e05964b108042b3cbd09d7faa4f6177e84914674180d9f2c64e299a7d6"}
Nov 01 10:30:10.341032 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.340992    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:10.442850 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.442761    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca\") pod \"183ae060-2054-42ff-ac8d-c674b2315c37\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") "
Nov 01 10:30:10.442850 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.442842    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh\" (UniqueName: \"kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh\") pod \"183ae060-2054-42ff-ac8d-c674b2315c37\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") "
Nov 01 10:30:10.443091 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.442925    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca\") pod \"183ae060-2054-42ff-ac8d-c674b2315c37\" (UID: \"183ae060-2054-42ff-ac8d-c674b2315c37\") "
Nov 01 10:30:10.443681 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.443643    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca" pod "183ae060-2054-42ff-ac8d-c674b2315c37" (UID: "183ae060-2054-42ff-ac8d-c674b2315c37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:30:10.446054 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.446020    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca" pod "183ae060-2054-42ff-ac8d-c674b2315c37" (UID: "183ae060-2054-42ff-ac8d-c674b2315c37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:30:10.446696 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.446656    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh" pod "183ae060-2054-42ff-ac8d-c674b2315c37" (UID: "183ae060-2054-42ff-ac8d-c674b2315c37"). InnerVolumeSpecName "kube-api-access-sspqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:30:10.543952 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.543913    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:10.543952 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.543950    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspqh\" (UniqueName: \"kubernetes.io/projected/183ae060-2054-42ff-ac8d-c674b2315c37-kube-api-access-sspqh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:10.544219 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:10.543967    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183ae060-2054-42ff-ac8d-c674b2315c37-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:11.174324 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:11.172929    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c" event={"ID":"183ae060-2054-42ff-ac8d-c674b2315c37","Type":"ContainerDied","Data":"706dc4dec1af94643661a352e5fa73920fdc5c67ec966fdac3fc07e33f8115b7"}
Nov 01 10:30:11.174324 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:11.172994    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706dc4dec1af94643661a352e5fa73920fdc5c67ec966fdac3fc07e33f8115b7"
Nov 01 10:30:11.174324 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:11.173069    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a1178-49g8c"
Nov 01 10:30:11.508556 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:30:11.503161    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:30:18.256487 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.256447    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"]
Nov 01 10:30:18.256913 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.256856    4314 cpu_manager.go:414] "containerMap: removing container" podUID="183ae060-2054-42ff-ac8d-c674b2315c37" containerName="revoke-keys"
Nov 01 10:30:18.256913 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.256868    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183ae060-2054-42ff-ac8d-c674b2315c37" containerName="revoke-keys"
Nov 01 10:30:18.256998 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.256925    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="183ae060-2054-42ff-ac8d-c674b2315c37" containerName="revoke-keys"
Nov 01 10:30:18.260605 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.260585    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.311202 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.311180    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:30:18.314803 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.314777    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"]
Nov 01 10:30:18.318631 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.318607    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvc2\" (UniqueName: \"kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.318758 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.318669    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.318826 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.318781    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.357000 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.356978    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:30:18.357108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.357001    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:30:18.357108 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.357064    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:30:18.419804 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.419770    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvc2\" (UniqueName: \"kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.419986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.419824    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.419986 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.419867    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.420465 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.420442    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.422192 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.422162    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.477668 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.477636    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvc2\" (UniqueName: \"kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2\") pod \"maas-api-revoke-sub-be11acb7-nfvq8\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.573079 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.573020    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:18.937954 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:18.937926    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"]
Nov 01 10:30:18.939116 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:30:18.939088    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90db4eda_6343_46f2_812a_7a72edd8e12d.slice/crio-74262bdfff87b1ff483bcf6e69f66a44a4e9153f8c62afee44321d4e13ca1b67 WatchSource:0}: Error finding container 74262bdfff87b1ff483bcf6e69f66a44a4e9153f8c62afee44321d4e13ca1b67: Status 404 returned error can't find the container with id 74262bdfff87b1ff483bcf6e69f66a44a4e9153f8c62afee44321d4e13ca1b67
Nov 01 10:30:19.259640 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:19.259608    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8" event={"ID":"90db4eda-6343-46f2-812a-7a72edd8e12d","Type":"ContainerStarted","Data":"54ca6fb08b21ad06cdacc263c0d8e69a579d54415c96cf7ca4a89d7d610baf09"}
Nov 01 10:30:19.260035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:19.259657    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8" event={"ID":"90db4eda-6343-46f2-812a-7a72edd8e12d","Type":"ContainerStarted","Data":"74262bdfff87b1ff483bcf6e69f66a44a4e9153f8c62afee44321d4e13ca1b67"}
Nov 01 10:30:19.369855 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:19.369811    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8" podStartSLOduration=1.369798382 podStartE2EDuration="1.369798382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:30:19.367965834 +0000 UTC m=+7092.562360389" watchObservedRunningTime="2026-11-01 10:30:19.369798382 +0000 UTC m=+7092.564192973"
Nov 01 10:30:20.266047 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:20.266014    4314 generic.go:356] "Generic (PLEG): container finished" podID="90db4eda-6343-46f2-812a-7a72edd8e12d" containerID="54ca6fb08b21ad06cdacc263c0d8e69a579d54415c96cf7ca4a89d7d610baf09" exitCode=0
Nov 01 10:30:20.266480 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:20.266107    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8" event={"ID":"90db4eda-6343-46f2-812a-7a72edd8e12d","Type":"ContainerDied","Data":"54ca6fb08b21ad06cdacc263c0d8e69a579d54415c96cf7ca4a89d7d610baf09"}
Nov 01 10:30:21.464159 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.464126    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:21.548768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.548298    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca\") pod \"90db4eda-6343-46f2-812a-7a72edd8e12d\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") "
Nov 01 10:30:21.548768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.548458    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2\" (UniqueName: \"kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2\") pod \"90db4eda-6343-46f2-812a-7a72edd8e12d\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") "
Nov 01 10:30:21.548768 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.548590    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca\") pod \"90db4eda-6343-46f2-812a-7a72edd8e12d\" (UID: \"90db4eda-6343-46f2-812a-7a72edd8e12d\") "
Nov 01 10:30:21.549264 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.549237    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca" pod "90db4eda-6343-46f2-812a-7a72edd8e12d" (UID: "90db4eda-6343-46f2-812a-7a72edd8e12d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:30:21.551015 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.550977    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca" pod "90db4eda-6343-46f2-812a-7a72edd8e12d" (UID: "90db4eda-6343-46f2-812a-7a72edd8e12d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:30:21.551472 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.551445    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2" pod "90db4eda-6343-46f2-812a-7a72edd8e12d" (UID: "90db4eda-6343-46f2-812a-7a72edd8e12d"). InnerVolumeSpecName "kube-api-access-hhvc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:30:21.649860 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.649821    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhvc2\" (UniqueName: \"kubernetes.io/projected/90db4eda-6343-46f2-812a-7a72edd8e12d-kube-api-access-hhvc2\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:21.649860 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.649856    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:21.650144 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:21.649872    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90db4eda-6343-46f2-812a-7a72edd8e12d-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:22.291247 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:22.291208    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8" event={"ID":"90db4eda-6343-46f2-812a-7a72edd8e12d","Type":"ContainerDied","Data":"74262bdfff87b1ff483bcf6e69f66a44a4e9153f8c62afee44321d4e13ca1b67"}
Nov 01 10:30:22.291566 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:22.291520    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74262bdfff87b1ff483bcf6e69f66a44a4e9153f8c62afee44321d4e13ca1b67"
Nov 01 10:30:22.291774 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:22.291758    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be11acb7-nfvq8"
Nov 01 10:30:29.051676 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.051644    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"]
Nov 01 10:30:29.052059 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.052035    4314 cpu_manager.go:414] "containerMap: removing container" podUID="90db4eda-6343-46f2-812a-7a72edd8e12d" containerName="revoke-keys"
Nov 01 10:30:29.052059 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.052045    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90db4eda-6343-46f2-812a-7a72edd8e12d" containerName="revoke-keys"
Nov 01 10:30:29.052141 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.052106    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="90db4eda-6343-46f2-812a-7a72edd8e12d" containerName="revoke-keys"
Nov 01 10:30:29.055296 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.055271    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.105664 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.105638    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:30:29.151575 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.151545    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:30:29.151575 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.151560    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:30:29.151781 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.151557    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:30:29.154407 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.154390    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"]
Nov 01 10:30:29.165060 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.165034    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.165172 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.165085    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzr8\" (UniqueName: \"kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.165225 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.165175    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.266478 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.266440    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzr8\" (UniqueName: \"kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.266705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.266492    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.266705 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.266580    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.267160 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.267135    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.268814 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.268796    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.324456 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.324364    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzr8\" (UniqueName: \"kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8\") pod \"maas-api-revoke-sub-fcd6ad1e-9gj9c\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.368065 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.368030    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:29.739885 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:29.739862    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"]
Nov 01 10:30:29.741267 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:30:29.741237    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50838726_4b4c_40b1_9e35_caebf2758514.slice/crio-95aad8cb2bf744ce4228aa9d0e394db6e6da40c44d3b7e6bb8bba33e2fc72568 WatchSource:0}: Error finding container 95aad8cb2bf744ce4228aa9d0e394db6e6da40c44d3b7e6bb8bba33e2fc72568: Status 404 returned error can't find the container with id 95aad8cb2bf744ce4228aa9d0e394db6e6da40c44d3b7e6bb8bba33e2fc72568
Nov 01 10:30:30.342378 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:30.342284    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c" event={"ID":"50838726-4b4c-40b1-9e35-caebf2758514","Type":"ContainerStarted","Data":"b838cff85ad91870746fff7af4c114d3ec5b45aa37a25e613e3511db888638c3"}
Nov 01 10:30:30.342378 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:30.342348    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c" event={"ID":"50838726-4b4c-40b1-9e35-caebf2758514","Type":"ContainerStarted","Data":"95aad8cb2bf744ce4228aa9d0e394db6e6da40c44d3b7e6bb8bba33e2fc72568"}
Nov 01 10:30:30.453083 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:30.453028    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c" podStartSLOduration=1.453011856 podStartE2EDuration="1.453011856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:30:30.451733669 +0000 UTC m=+7103.646128225" watchObservedRunningTime="2026-11-01 10:30:30.453011856 +0000 UTC m=+7103.647406410"
Nov 01 10:30:31.349098 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:31.349067    4314 generic.go:356] "Generic (PLEG): container finished" podID="50838726-4b4c-40b1-9e35-caebf2758514" containerID="b838cff85ad91870746fff7af4c114d3ec5b45aa37a25e613e3511db888638c3" exitCode=0
Nov 01 10:30:31.349606 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:31.349156    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c" event={"ID":"50838726-4b4c-40b1-9e35-caebf2758514","Type":"ContainerDied","Data":"b838cff85ad91870746fff7af4c114d3ec5b45aa37a25e613e3511db888638c3"}
Nov 01 10:30:32.548107 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.548076    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:32.603415 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.603374    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca\") pod \"50838726-4b4c-40b1-9e35-caebf2758514\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") "
Nov 01 10:30:32.603623 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.603518    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8\" (UniqueName: \"kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8\") pod \"50838726-4b4c-40b1-9e35-caebf2758514\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") "
Nov 01 10:30:32.604015 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.603847    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca" pod "50838726-4b4c-40b1-9e35-caebf2758514" (UID: "50838726-4b4c-40b1-9e35-caebf2758514"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:30:32.604158 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.604008    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca\") pod \"50838726-4b4c-40b1-9e35-caebf2758514\" (UID: \"50838726-4b4c-40b1-9e35-caebf2758514\") "
Nov 01 10:30:32.604653 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.604628    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:32.606401 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.606373    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8" pod "50838726-4b4c-40b1-9e35-caebf2758514" (UID: "50838726-4b4c-40b1-9e35-caebf2758514"). InnerVolumeSpecName "kube-api-access-kdzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:30:32.607210 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.607189    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca" pod "50838726-4b4c-40b1-9e35-caebf2758514" (UID: "50838726-4b4c-40b1-9e35-caebf2758514"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:30:32.747559 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.724196    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50838726-4b4c-40b1-9e35-caebf2758514-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:32.747867 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:32.747837    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdzr8\" (UniqueName: \"kubernetes.io/projected/50838726-4b4c-40b1-9e35-caebf2758514-kube-api-access-kdzr8\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:33.384614 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:33.383775    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c" event={"ID":"50838726-4b4c-40b1-9e35-caebf2758514","Type":"ContainerDied","Data":"95aad8cb2bf744ce4228aa9d0e394db6e6da40c44d3b7e6bb8bba33e2fc72568"}
Nov 01 10:30:33.384614 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:33.383840    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95aad8cb2bf744ce4228aa9d0e394db6e6da40c44d3b7e6bb8bba33e2fc72568"
Nov 01 10:30:33.384614 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:33.383915    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcd6ad1e-9gj9c"
Nov 01 10:30:34.800563 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:34.799783    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-6qxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:30:34.800563 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:34.799866    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-6qxcm" podUID="2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:30:35.051607 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:35.050836    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"]
Nov 01 10:30:35.105748 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:35.105713    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7abfbdf-wkgfl"]
Nov 01 10:30:36.926823 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:36.926792    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fdadd0-0371-4f60-b22c-74e711307199" path="/var/lib/kubelet/pods/69fdadd0-0371-4f60-b22c-74e711307199/volumes"
Nov 01 10:30:39.782513 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.782482    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"]
Nov 01 10:30:39.782994 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.782883    4314 cpu_manager.go:414] "containerMap: removing container" podUID="50838726-4b4c-40b1-9e35-caebf2758514" containerName="revoke-keys"
Nov 01 10:30:39.782994 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.782895    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50838726-4b4c-40b1-9e35-caebf2758514" containerName="revoke-keys"
Nov 01 10:30:39.782994 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.782979    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="50838726-4b4c-40b1-9e35-caebf2758514" containerName="revoke-keys"
Nov 01 10:30:39.786285 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.786268    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.830919 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.830892    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.831042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.830930    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwphw\" (UniqueName: \"kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.831042 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.831024    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.837071 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.837040    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:30:39.837509 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.837487    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:30:39.883051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.883027    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:30:39.883051 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.883030    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:30:39.885936 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.885915    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"]
Nov 01 10:30:39.932100 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.932074    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.932261 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.932106    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwphw\" (UniqueName: \"kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.932261 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.932129    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.932794 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.932772    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.934553 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.934514    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:39.989989 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:39.989951    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwphw\" (UniqueName: \"kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw\") pod \"maas-api-revoke-sub-de8ad5db-dz6fv\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:40.098837 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:40.098746    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:40.465363 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:40.465334    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"]
Nov 01 10:30:40.466692 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:30:40.466669    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a20119_c12a_4331_9e3c_fb014888bec8.slice/crio-ed3453c6c7eaffda78c8366638196fe024e7b04ea01ab6bffa1b9b16432add26 WatchSource:0}: Error finding container ed3453c6c7eaffda78c8366638196fe024e7b04ea01ab6bffa1b9b16432add26: Status 404 returned error can't find the container with id ed3453c6c7eaffda78c8366638196fe024e7b04ea01ab6bffa1b9b16432add26
Nov 01 10:30:40.955215 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:40.955183    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:30:41.323154 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:30:41.323114    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:30:41.444577 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:41.444516    4314 generic.go:356] "Generic (PLEG): container finished" podID="58a20119-c12a-4331-9e3c-fb014888bec8" containerID="2eacfb4b832b345e273cf811cf8fa3d35fbb5dc28ea462c411ba1585ef457072" exitCode=0
Nov 01 10:30:41.444758 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:41.444602    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv" event={"ID":"58a20119-c12a-4331-9e3c-fb014888bec8","Type":"ContainerDied","Data":"2eacfb4b832b345e273cf811cf8fa3d35fbb5dc28ea462c411ba1585ef457072"}
Nov 01 10:30:41.444758 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:41.444658    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv" event={"ID":"58a20119-c12a-4331-9e3c-fb014888bec8","Type":"ContainerStarted","Data":"ed3453c6c7eaffda78c8366638196fe024e7b04ea01ab6bffa1b9b16432add26"}
Nov 01 10:30:42.651169 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.651131    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:42.760774 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.760713    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca\") pod \"58a20119-c12a-4331-9e3c-fb014888bec8\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") "
Nov 01 10:30:42.761012 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.760852    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca\") pod \"58a20119-c12a-4331-9e3c-fb014888bec8\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") "
Nov 01 10:30:42.761012 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.760909    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw\" (UniqueName: \"kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw\") pod \"58a20119-c12a-4331-9e3c-fb014888bec8\" (UID: \"58a20119-c12a-4331-9e3c-fb014888bec8\") "
Nov 01 10:30:42.762044 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.761992    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca" pod "58a20119-c12a-4331-9e3c-fb014888bec8" (UID: "58a20119-c12a-4331-9e3c-fb014888bec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:30:42.764524 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.764495    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw" pod "58a20119-c12a-4331-9e3c-fb014888bec8" (UID: "58a20119-c12a-4331-9e3c-fb014888bec8"). InnerVolumeSpecName "kube-api-access-dwphw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:30:42.765266 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.765223    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca" pod "58a20119-c12a-4331-9e3c-fb014888bec8" (UID: "58a20119-c12a-4331-9e3c-fb014888bec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:30:42.862063 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.862019    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:42.862063 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.862065    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a20119-c12a-4331-9e3c-fb014888bec8-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:42.862374 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:42.862081    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwphw\" (UniqueName: \"kubernetes.io/projected/58a20119-c12a-4331-9e3c-fb014888bec8-kube-api-access-dwphw\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:43.557362 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:43.557319    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv" event={"ID":"58a20119-c12a-4331-9e3c-fb014888bec8","Type":"ContainerDied","Data":"ed3453c6c7eaffda78c8366638196fe024e7b04ea01ab6bffa1b9b16432add26"}
Nov 01 10:30:43.561576 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:43.557740    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3453c6c7eaffda78c8366638196fe024e7b04ea01ab6bffa1b9b16432add26"
Nov 01 10:30:43.561927 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:43.559699    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de8ad5db-dz6fv"
Nov 01 10:30:45.075845 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:45.075810    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"]
Nov 01 10:30:45.117993 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:45.117963    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e0baa9-g4v5n"]
Nov 01 10:30:46.926667 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:46.926634    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500d094f-6f86-46ec-8869-8372d772e308" path="/var/lib/kubelet/pods/500d094f-6f86-46ec-8869-8372d772e308/volumes"
Nov 01 10:30:50.557450 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.557408    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"]
Nov 01 10:30:50.557965 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.557926    4314 cpu_manager.go:414] "containerMap: removing container" podUID="58a20119-c12a-4331-9e3c-fb014888bec8" containerName="revoke-keys"
Nov 01 10:30:50.557965 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.557939    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a20119-c12a-4331-9e3c-fb014888bec8" containerName="revoke-keys"
Nov 01 10:30:50.558061 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.558017    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a20119-c12a-4331-9e3c-fb014888bec8" containerName="revoke-keys"
Nov 01 10:30:50.562721 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.562700    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.614116 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.614078    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"]
Nov 01 10:30:50.657035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.657000    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:30:50.657035 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.657000    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:30:50.657263 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.657053    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:30:50.657734 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.657718    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:30:50.722520 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.722477    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.722735 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.722555    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.722735 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.722617    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqrh\" (UniqueName: \"kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.823836 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.823754    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.823836 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.823816    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.824055 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.823857    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqrh\" (UniqueName: \"kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.824659 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.824633    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.826190 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.826171    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:50.881919 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:50.881883    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqrh\" (UniqueName: \"kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh\") pod \"maas-api-revoke-sub-300abff5-rjdpm\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:51.178333 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:51.178249    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:51.543809 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:51.543783    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"]
Nov 01 10:30:51.545384 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:30:51.545356    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7aacf8_ee82_400f_a47e_536d440956db.slice/crio-b5e2e982ab8d4eee91ae227ff77181f22f6cdbcedd59f77b94d7d46724f74ba3 WatchSource:0}: Error finding container b5e2e982ab8d4eee91ae227ff77181f22f6cdbcedd59f77b94d7d46724f74ba3: Status 404 returned error can't find the container with id b5e2e982ab8d4eee91ae227ff77181f22f6cdbcedd59f77b94d7d46724f74ba3
Nov 01 10:30:51.630131 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:51.630092    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm" event={"ID":"fc7aacf8-ee82-400f-a47e-536d440956db","Type":"ContainerStarted","Data":"b5e2e982ab8d4eee91ae227ff77181f22f6cdbcedd59f77b94d7d46724f74ba3"}
Nov 01 10:30:52.638154 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:52.638115    4314 generic.go:356] "Generic (PLEG): container finished" podID="fc7aacf8-ee82-400f-a47e-536d440956db" containerID="12ddc34872fe78f45e5afd0424e6060be024c0338703587bbd08f5c46ef179eb" exitCode=0
Nov 01 10:30:52.638660 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:52.638194    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm" event={"ID":"fc7aacf8-ee82-400f-a47e-536d440956db","Type":"ContainerDied","Data":"12ddc34872fe78f45e5afd0424e6060be024c0338703587bbd08f5c46ef179eb"}
Nov 01 10:30:53.846197 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.846163    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:30:53.960125 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.959713    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca\") pod \"fc7aacf8-ee82-400f-a47e-536d440956db\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") "
Nov 01 10:30:53.960125 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.959764    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca\") pod \"fc7aacf8-ee82-400f-a47e-536d440956db\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") "
Nov 01 10:30:53.960125 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.959846    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh\" (UniqueName: \"kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh\") pod \"fc7aacf8-ee82-400f-a47e-536d440956db\" (UID: \"fc7aacf8-ee82-400f-a47e-536d440956db\") "
Nov 01 10:30:53.967072 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.967026    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca" pod "fc7aacf8-ee82-400f-a47e-536d440956db" (UID: "fc7aacf8-ee82-400f-a47e-536d440956db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:30:53.969692 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.969651    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca" pod "fc7aacf8-ee82-400f-a47e-536d440956db" (UID: "fc7aacf8-ee82-400f-a47e-536d440956db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:30:53.975697 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:53.975660    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh" pod "fc7aacf8-ee82-400f-a47e-536d440956db" (UID: "fc7aacf8-ee82-400f-a47e-536d440956db"). InnerVolumeSpecName "kube-api-access-hmqrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:30:54.061344 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:54.061300    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:54.061344 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:54.061339    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc7aacf8-ee82-400f-a47e-536d440956db-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:54.061662 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:54.061357    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqrh\" (UniqueName: \"kubernetes.io/projected/fc7aacf8-ee82-400f-a47e-536d440956db-kube-api-access-hmqrh\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:30:54.661954 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:54.661917    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm" event={"ID":"fc7aacf8-ee82-400f-a47e-536d440956db","Type":"ContainerDied","Data":"b5e2e982ab8d4eee91ae227ff77181f22f6cdbcedd59f77b94d7d46724f74ba3"}
Nov 01 10:30:54.662251 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:54.662232    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e2e982ab8d4eee91ae227ff77181f22f6cdbcedd59f77b94d7d46724f74ba3"
Nov 01 10:30:54.662441 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:30:54.662426    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300abff5-rjdpm"
Nov 01 10:31:01.288674 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.288635    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"]
Nov 01 10:31:01.289201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.289120    4314 cpu_manager.go:414] "containerMap: removing container" podUID="fc7aacf8-ee82-400f-a47e-536d440956db" containerName="revoke-keys"
Nov 01 10:31:01.289201 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.289133    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7aacf8-ee82-400f-a47e-536d440956db" containerName="revoke-keys"
Nov 01 10:31:01.289289 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.289223    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7aacf8-ee82-400f-a47e-536d440956db" containerName="revoke-keys"
Nov 01 10:31:01.292733 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.292718    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.342423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.342341    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 10:31:01.342423 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.342387    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:31:01.388422 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.388392    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 10:31:01.388630 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.388440    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\""
Nov 01 10:31:01.389979 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.389961    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.390067 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.389992    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.390067 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.390011    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflsp\" (UniqueName: \"kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.391470 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.391451    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"]
Nov 01 10:31:01.491637 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.491598    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.491831 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.491648    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.491831 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.491666    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflsp\" (UniqueName: \"kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.492439 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.492417    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.494019 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.493994    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.549821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.549750    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflsp\" (UniqueName: \"kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp\") pod \"maas-api-revoke-sub-06e1f86b-gbm6d\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.605618 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.605574    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:01.975132 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:01.975107    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"]
Nov 01 10:31:01.976454 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:31:01.976418    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90252d8_8c11_4e66_8871_1163b572d48b.slice/crio-475377c797364a183133248de6769a61129b1121f62ee81ee760f54bf39cd3a9 WatchSource:0}: Error finding container 475377c797364a183133248de6769a61129b1121f62ee81ee760f54bf39cd3a9: Status 404 returned error can't find the container with id 475377c797364a183133248de6769a61129b1121f62ee81ee760f54bf39cd3a9
Nov 01 10:31:02.738020 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:02.737890    4314 generic.go:356] "Generic (PLEG): container finished" podID="a90252d8-8c11-4e66-8871-1163b572d48b" containerID="0ac4f590caad8a8103ceee30791205fcb78bf7483c4643491a3cea9040c0bfd0" exitCode=0
Nov 01 10:31:02.738020 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:02.737937    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d" event={"ID":"a90252d8-8c11-4e66-8871-1163b572d48b","Type":"ContainerDied","Data":"0ac4f590caad8a8103ceee30791205fcb78bf7483c4643491a3cea9040c0bfd0"}
Nov 01 10:31:02.738020 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:02.737993    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d" event={"ID":"a90252d8-8c11-4e66-8871-1163b572d48b","Type":"ContainerStarted","Data":"475377c797364a183133248de6769a61129b1121f62ee81ee760f54bf39cd3a9"}
Nov 01 10:31:03.951916 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:03.951660    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:04.137621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.136074    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp\" (UniqueName: \"kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp\") pod \"a90252d8-8c11-4e66-8871-1163b572d48b\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") "
Nov 01 10:31:04.137621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.136137    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca\") pod \"a90252d8-8c11-4e66-8871-1163b572d48b\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") "
Nov 01 10:31:04.137621 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.136184    4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca\") pod \"a90252d8-8c11-4e66-8871-1163b572d48b\" (UID: \"a90252d8-8c11-4e66-8871-1163b572d48b\") "
Nov 01 10:31:04.144290 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.144255    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca" pod "a90252d8-8c11-4e66-8871-1163b572d48b" (UID: "a90252d8-8c11-4e66-8871-1163b572d48b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:31:04.256560 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.256218    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-openshift-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:31:04.320586 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.315795    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca" pod "a90252d8-8c11-4e66-8871-1163b572d48b" (UID: "a90252d8-8c11-4e66-8871-1163b572d48b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:31:04.346564 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.345780    4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp" pod "a90252d8-8c11-4e66-8871-1163b572d48b" (UID: "a90252d8-8c11-4e66-8871-1163b572d48b"). InnerVolumeSpecName "kube-api-access-wflsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:31:04.362554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.360710    4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90252d8-8c11-4e66-8871-1163b572d48b-maas-api-xks-ca\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:31:04.362554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.360760    4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflsp\" (UniqueName: \"kubernetes.io/projected/a90252d8-8c11-4e66-8871-1163b572d48b-kube-api-access-wflsp\") on node \"ip-10-0-134-93.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:31:04.755792 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.755748    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d" event={"ID":"a90252d8-8c11-4e66-8871-1163b572d48b","Type":"ContainerDied","Data":"475377c797364a183133248de6769a61129b1121f62ee81ee760f54bf39cd3a9"}
Nov 01 10:31:04.756093 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.756072    4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="475377c797364a183133248de6769a61129b1121f62ee81ee760f54bf39cd3a9"
Nov 01 10:31:04.756313 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:04.756287    4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e1f86b-gbm6d"
Nov 01 10:31:11.321967 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:31:11.321929    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:31:13.307547 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:13.307443    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:31:17.145159 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:17.145118    4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-n4qzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:31:17.145844 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:17.145808    4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-n4qzc" podUID="4cc4fee6-5729-487e-9ae7-e7997f9fcb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:31:22.048280 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:22.048202    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"]
Nov 01 10:31:22.094719 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:22.094690    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b61a82-7vxwt"]
Nov 01 10:31:22.927198 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:22.927168    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a2cc7f-015f-4b53-8b50-b17283bbba9b" path="/var/lib/kubelet/pods/24a2cc7f-015f-4b53-8b50-b17283bbba9b/volumes"
Nov 01 10:31:41.213284 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:41.213249    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8jqs6_9f09aaaf-f882-4e11-b764-bf87a9351d3e/manager/0.log"
Nov 01 10:31:41.321784 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:31:41.321746    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:31:41.325357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:41.325321    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:31:41.787724 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:41.787694    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d89f6bbc8-z9hs6_5d16903d-3097-4cab-8cdb-fe2491d19902/manager/0.log"
Nov 01 10:31:44.042057 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:44.042030    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-06e1f86b-gbm6d_a90252d8-8c11-4e66-8871-1163b572d48b/revoke-keys/0.log"
Nov 01 10:31:44.257297 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:44.257273    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-136a1178-49g8c_183ae060-2054-42ff-ac8d-c674b2315c37/revoke-keys/0.log"
Nov 01 10:31:44.462357 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:44.462283    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-21da78ef-fft7c_17c5f0dc-ec2d-4e12-875d-189df8f0e9b0/revoke-keys/0.log"
Nov 01 10:31:44.686913 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:44.686882    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-300abff5-rjdpm_fc7aacf8-ee82-400f-a47e-536d440956db/revoke-keys/0.log"
Nov 01 10:31:44.907189 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:44.907159    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-47c1d6cd-jrswv_40f2ea1c-d56c-40f5-b34d-171c7210fcd4/revoke-keys/0.log"
Nov 01 10:31:45.124064 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:45.124032    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-61f83c3e-bpj7p_dc85bc4b-a2ba-4c86-90f8-6bf795b142c3/revoke-keys/0.log"
Nov 01 10:31:45.340399 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:45.340370    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d14c8bb-zzhcg_4ccf1125-1922-4898-94f9-6dbfe0b3b1d8/revoke-keys/0.log"
Nov 01 10:31:45.566696 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:45.566662    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8e5631b3-tlkpw_ac980a5f-f55d-4452-bd4d-4de7f586237f/revoke-keys/0.log"
Nov 01 10:31:45.793859 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:45.793829    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a15030fc-4fw6l_730b33a3-b3e3-4c83-bd68-977d1cf50500/revoke-keys/0.log"
Nov 01 10:31:46.013803 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:46.013777    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ad11aab6-69llm_0047fc77-ce14-4867-821b-aca7ae3254a9/revoke-keys/0.log"
Nov 01 10:31:46.227547 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:46.227455    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-be11acb7-nfvq8_90db4eda-6343-46f2-812a-7a72edd8e12d/revoke-keys/0.log"
Nov 01 10:31:46.447048 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:46.447014    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d42f98b8-gxc29_eaa5ce07-d1e0-447e-a289-ad761d029a40/revoke-keys/0.log"
Nov 01 10:31:46.669182 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:46.669150    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-de8ad5db-dz6fv_58a20119-c12a-4331-9e3c-fb014888bec8/revoke-keys/0.log"
Nov 01 10:31:46.879459 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:46.879419    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e539d9ea-tgvgw_095a6e12-7129-4eee-9842-d74276ec0441/revoke-keys/0.log"
Nov 01 10:31:47.103923 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:47.103896    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee2f6cad-9f4rf_b8ecbe33-82b5-4948-90dd-5c56b20ee755/revoke-keys/0.log"
Nov 01 10:31:47.327373 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:47.327345    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fcd6ad1e-9gj9c_50838726-4b4c-40b1-9e35-caebf2758514/revoke-keys/0.log"
Nov 01 10:31:50.383060 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:50.383031    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-h6b62_f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad/kuadrant-console-plugin/0.log"
Nov 01 10:31:51.188174 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:51.188146    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-l9ctz_f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e/manager/0.log"
Nov 01 10:31:53.122899 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:53.122865    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649b5c55d-mck42_2d3a6367-0131-48e9-b825-19cf48b23b87/kube-auth-proxy/0.log"
Nov 01 10:31:53.826698 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:53.826672    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-6qxcm_2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6/istio-proxy/0.log"
Nov 01 10:31:54.072434 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:54.072404    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bx5ck_d68734d8-e154-4730-990c-3bb7292efaa0/istio-proxy/0.log"
Nov 01 10:31:54.537554 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:54.537513    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-h8njt_3959e93c-0abf-409a-9f46-d31930d382c6/istio-proxy/0.log"
Nov 01 10:31:55.008683 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:55.008652    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-n4qzc_4cc4fee6-5729-487e-9ae7-e7997f9fcb3e/istio-proxy/0.log"
Nov 01 10:31:56.125275 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:56.125250    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-797f57964-nlqd8_3415ad8e-a2dd-4804-9eff-7145c8fa8678/payload-processing/0.log"
Nov 01 10:31:58.099224 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:58.099186    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xcr5m_3855ed68-9c33-4e39-b69a-c56f27fa6d46/main/0.log"
Nov 01 10:31:58.547017 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:58.546988    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-866l2_cc3d132e-8479-4c24-a5a1-8dcc1d404880/main/0.log"
Nov 01 10:31:58.992584 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:31:58.992556    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-v58np_52a83d0f-f6cd-4498-9982-6dae24b6639b/main/0.log"
Nov 01 10:32:06.046909 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:06.046873    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"]
Nov 01 10:32:06.092595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:06.092569    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42f98b8-gxc29"]
Nov 01 10:32:06.927401 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:06.927367    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa5ce07-d1e0-447e-a289-ad761d029a40" path="/var/lib/kubelet/pods/eaa5ce07-d1e0-447e-a289-ad761d029a40/volumes"
Nov 01 10:32:08.804156 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.804123    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mp8j/must-gather-vxb74"]
Nov 01 10:32:08.804731 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.804711    4314 cpu_manager.go:414] "containerMap: removing container" podUID="a90252d8-8c11-4e66-8871-1163b572d48b" containerName="revoke-keys"
Nov 01 10:32:08.804821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.804733    4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90252d8-8c11-4e66-8871-1163b572d48b" containerName="revoke-keys"
Nov 01 10:32:08.804897 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.804882    4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90252d8-8c11-4e66-8871-1163b572d48b" containerName="revoke-keys"
Nov 01 10:32:08.808375 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.808353    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:08.864124 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.862609    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mp8j/must-gather-vxb74"]
Nov 01 10:32:08.868449 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.868422    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:32:08.904904 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.904872    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5mp8j\"/\"default-dockercfg-w4vsc\""
Nov 01 10:32:08.904904 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.904872    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mp8j\"/\"openshift-service-ca.crt\""
Nov 01 10:32:08.905068 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.904915    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mp8j\"/\"kube-root-ca.crt\""
Nov 01 10:32:08.962446 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.962424    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkprd\" (UniqueName: \"kubernetes.io/projected/a2cbbebb-6022-4aa4-8b71-28e449e26f4e-kube-api-access-lkprd\") pod \"must-gather-vxb74\" (UID: \"a2cbbebb-6022-4aa4-8b71-28e449e26f4e\") " pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:08.962595 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:08.962455    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2cbbebb-6022-4aa4-8b71-28e449e26f4e-must-gather-output\") pod \"must-gather-vxb74\" (UID: \"a2cbbebb-6022-4aa4-8b71-28e449e26f4e\") " pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:09.043937 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.043910    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:32:09.063479 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.063419    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkprd\" (UniqueName: \"kubernetes.io/projected/a2cbbebb-6022-4aa4-8b71-28e449e26f4e-kube-api-access-lkprd\") pod \"must-gather-vxb74\" (UID: \"a2cbbebb-6022-4aa4-8b71-28e449e26f4e\") " pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:09.063479 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.063450    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2cbbebb-6022-4aa4-8b71-28e449e26f4e-must-gather-output\") pod \"must-gather-vxb74\" (UID: \"a2cbbebb-6022-4aa4-8b71-28e449e26f4e\") " pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:09.063839 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.063816    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2cbbebb-6022-4aa4-8b71-28e449e26f4e-must-gather-output\") pod \"must-gather-vxb74\" (UID: \"a2cbbebb-6022-4aa4-8b71-28e449e26f4e\") " pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:09.123322 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.123294    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkprd\" (UniqueName: \"kubernetes.io/projected/a2cbbebb-6022-4aa4-8b71-28e449e26f4e-kube-api-access-lkprd\") pod \"must-gather-vxb74\" (UID: \"a2cbbebb-6022-4aa4-8b71-28e449e26f4e\") " pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:09.422567 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.422469    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mp8j/must-gather-vxb74"
Nov 01 10:32:09.788217 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.788191    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mp8j/must-gather-vxb74"]
Nov 01 10:32:09.789744 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:32:09.789717    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2cbbebb_6022_4aa4_8b71_28e449e26f4e.slice/crio-2a9304c028bb00023aa7dc86598807e7a3cb161e25bc0c0c801367870edac602 WatchSource:0}: Error finding container 2a9304c028bb00023aa7dc86598807e7a3cb161e25bc0c0c801367870edac602: Status 404 returned error can't find the container with id 2a9304c028bb00023aa7dc86598807e7a3cb161e25bc0c0c801367870edac602
Nov 01 10:32:09.791431 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:09.791411    4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 10:32:10.182312 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:10.182234    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mp8j/must-gather-vxb74" event={"ID":"a2cbbebb-6022-4aa4-8b71-28e449e26f4e","Type":"ContainerStarted","Data":"2a9304c028bb00023aa7dc86598807e7a3cb161e25bc0c0c801367870edac602"}
Nov 01 10:32:11.321273 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:32:11.321241    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:32:12.019118 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:12.019086    4314 scope.go:122] "RemoveContainer" containerID="1cf33624bcf2e7d062a58fd18bd8b1e47564b271041b43afbb81d6f680494cee"
Nov 01 10:32:12.027910 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:12.027888    4314 scope.go:122] "RemoveContainer" containerID="9068ff7bb6ebaa9bc2ae08a0a26468dcd678dc30cee969212afdac70773ae272"
Nov 01 10:32:17.053483 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:17.053451    4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"]
Nov 01 10:32:17.065728 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:17.065703    4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee2f6cad-9f4rf"]
Nov 01 10:32:17.240754 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:17.240712    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mp8j/must-gather-vxb74" event={"ID":"a2cbbebb-6022-4aa4-8b71-28e449e26f4e","Type":"ContainerStarted","Data":"5b2d876903f300ea5e3085ee079d0a02e1cc8beae41b1f0bb1a72abd99d5dba5"}
Nov 01 10:32:17.240930 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:17.240764    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mp8j/must-gather-vxb74" event={"ID":"a2cbbebb-6022-4aa4-8b71-28e449e26f4e","Type":"ContainerStarted","Data":"5eaea457a057447c4611e90a6360e8ed1fcfb979da52ae7b44b16e4bbbd46973"}
Nov 01 10:32:17.351888 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:17.351784    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5mp8j/must-gather-vxb74" podStartSLOduration=2.832644058 podStartE2EDuration="9.351768331s" totalImagesPullingTime="6.519124273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:32:17.348974415 +0000 UTC m=+7210.543368987" watchObservedRunningTime="2026-11-01 10:32:17.351768331 +0000 UTC m=+7210.546162889"
Nov 01 10:32:18.380439 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:18.380409    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5cgcj_3370a9e2-dba0-475a-a84e-3d0d1cfab280/global-pull-secret-syncer/0.log"
Nov 01 10:32:18.927300 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:18.927270    4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ecbe33-82b5-4948-90dd-5c56b20ee755" path="/var/lib/kubelet/pods/b8ecbe33-82b5-4948-90dd-5c56b20ee755/volumes"
Nov 01 10:32:20.788222 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:20.788191    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8jqs6_9f09aaaf-f882-4e11-b764-bf87a9351d3e/manager/0.log"
Nov 01 10:32:21.622435 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:21.622404    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log"
Nov 01 10:32:21.728376 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:21.728345    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d89f6bbc8-z9hs6_5d16903d-3097-4cab-8cdb-fe2491d19902/manager/0.log"
Nov 01 10:32:22.888815 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:22.888787    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-93.us-west-2.compute.internal_67982dcee5ed5a322620afe889b9e391/haproxy/0.log"
Nov 01 10:32:27.823967 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:27.823936    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8jqs6_9f09aaaf-f882-4e11-b764-bf87a9351d3e/manager/0.log"
Nov 01 10:32:28.782453 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.782422    4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"]
Nov 01 10:32:28.784526 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.784467    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d89f6bbc8-z9hs6_5d16903d-3097-4cab-8cdb-fe2491d19902/manager/0.log"
Nov 01 10:32:28.791327 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.791303    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.839662 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.839637    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"]
Nov 01 10:32:28.858807 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.858780    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-proc\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.858922 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.858837    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-lib-modules\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.858993 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.858966    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-podres\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.859045 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.859031    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8lp\" (UniqueName: \"kubernetes.io/projected/f7d9857c-1764-467f-9ab8-97ded0289ec7-kube-api-access-7g8lp\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.859084 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.859053    4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-sys\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.882218 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.882195    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w96kq\"/\"kube-root-ca.crt\""
Nov 01 10:32:28.882326 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.882267    4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w96kq\"/\"default-dockercfg-f9ll4\""
Nov 01 10:32:28.885649 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.885632    4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w96kq\"/\"openshift-service-ca.crt\""
Nov 01 10:32:28.959910 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.959882    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-proc\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.959937    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-lib-modules\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.959964    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-podres\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.959991    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8lp\" (UniqueName: \"kubernetes.io/projected/f7d9857c-1764-467f-9ab8-97ded0289ec7-kube-api-access-7g8lp\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.959995    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-proc\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.960007    4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-sys\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.960022    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-lib-modules\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.960079    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-sys\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:28.960094 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:28.960080    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7d9857c-1764-467f-9ab8-97ded0289ec7-podres\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:29.018524 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:29.018496    4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8lp\" (UniqueName: \"kubernetes.io/projected/f7d9857c-1764-467f-9ab8-97ded0289ec7-kube-api-access-7g8lp\") pod \"perf-node-gather-daemonset-v4s95\" (UID: \"f7d9857c-1764-467f-9ab8-97ded0289ec7\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:29.106744 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:29.106682    4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:29.475162 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:29.475134    4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"]
Nov 01 10:32:29.477112 ip-10-0-134-93 kubenswrapper[4314]: W1101 10:32:29.477066    4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7d9857c_1764_467f_9ab8_97ded0289ec7.slice/crio-697dc451c5170cb25ac1d49ff5cae49b1d796fc7a4afda186e4e516b8ea696e3 WatchSource:0}: Error finding container 697dc451c5170cb25ac1d49ff5cae49b1d796fc7a4afda186e4e516b8ea696e3: Status 404 returned error can't find the container with id 697dc451c5170cb25ac1d49ff5cae49b1d796fc7a4afda186e4e516b8ea696e3
Nov 01 10:32:29.623248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:29.623228    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-h6b62_f96fe7be-b8d7-4c5d-bbbe-dd5b8143a4ad/kuadrant-console-plugin/0.log"
Nov 01 10:32:30.318365 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:30.318296    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95" event={"ID":"f7d9857c-1764-467f-9ab8-97ded0289ec7","Type":"ContainerStarted","Data":"0a4da0adcf655b0209b9808d99429926617f76b35ae732d348d5246cfe45a793"}
Nov 01 10:32:30.318365 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:30.318356    4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95" event={"ID":"f7d9857c-1764-467f-9ab8-97ded0289ec7","Type":"ContainerStarted","Data":"697dc451c5170cb25ac1d49ff5cae49b1d796fc7a4afda186e4e516b8ea696e3"}
Nov 01 10:32:30.318916 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:30.318486    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:30.428821 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:30.428765    4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95" podStartSLOduration=2.428746652 podStartE2EDuration="2.428746652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 10:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:32:30.426479354 +0000 UTC m=+7223.620873910" watchObservedRunningTime="2026-11-01 10:32:30.428746652 +0000 UTC m=+7223.623141207"
Nov 01 10:32:31.004422 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:31.004387    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-l9ctz_f0de96e5-dcbb-4a01-a4a9-10d3b4ec230e/manager/0.log"
Nov 01 10:32:34.633659 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:34.633631    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8jqs6_9f09aaaf-f882-4e11-b764-bf87a9351d3e/manager/0.log"
Nov 01 10:32:35.523248 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:35.523216    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d89f6bbc8-z9hs6_5d16903d-3097-4cab-8cdb-fe2491d19902/manager/0.log"
Nov 01 10:32:36.336054 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:36.336025    4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-v4s95"
Nov 01 10:32:38.329394 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:38.329358    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-646fcdc6b8-jpgtr_2fe94ec3-8a61-4a0e-be18-4a89125ffce8/metrics-server/0.log"
Nov 01 10:32:39.238024 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:39.237990    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8t4zj_52d7169a-99e2-45ce-a22b-4454cd4d7fc1/node-exporter/0.log"
Nov 01 10:32:39.662759 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:39.662694    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8t4zj_52d7169a-99e2-45ce-a22b-4454cd4d7fc1/kube-rbac-proxy/0.log"
Nov 01 10:32:40.089881 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:40.089853    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8t4zj_52d7169a-99e2-45ce-a22b-4454cd4d7fc1/init-textfile/0.log"
Nov 01 10:32:41.325293 ip-10-0-134-93 kubenswrapper[4314]: E1101 10:32:41.325255    4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4bc2" containerName="extractor"
Nov 01 10:32:41.648247 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:41.648148    4314 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-slw5x_bade5291-3729-42d3-92da-7fe9e224a4ad/konnectivity-agent/0.log" limit=1024
Nov 01 10:32:44.350648 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:44.350617    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/prometheus/0.log"
Nov 01 10:32:44.771226 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:44.771197    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/config-reloader/0.log"
Nov 01 10:32:45.192196 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:45.192116    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/thanos-sidecar/0.log"
Nov 01 10:32:45.612311 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:45.612267    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/kube-rbac-proxy-web/0.log"
Nov 01 10:32:46.006427 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:46.006400    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649b5c55d-mck42_2d3a6367-0131-48e9-b825-19cf48b23b87/kube-auth-proxy/0.log"
Nov 01 10:32:46.046433 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:46.046410    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/kube-rbac-proxy/0.log"
Nov 01 10:32:46.466741 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:46.466668    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/kube-rbac-proxy-thanos/0.log"
Nov 01 10:32:46.637548 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:46.637509    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-6qxcm_2e94d8d4-6cd1-4e92-b324-ae94d3bf49b6/istio-proxy/0.log"
Nov 01 10:32:46.844988 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:46.844958    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bx5ck_d68734d8-e154-4730-990c-3bb7292efaa0/istio-proxy/0.log"
Nov 01 10:32:46.888558 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:46.888515    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33087e46-8c7e-4d5a-a7d3-888da30bb31d/init-config-reloader/0.log"
Nov 01 10:32:47.276753 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:47.276725    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-h8njt_3959e93c-0abf-409a-9f46-d31930d382c6/istio-proxy/0.log"
Nov 01 10:32:47.730722 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:47.730687    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-n4qzc_4cc4fee6-5729-487e-9ae7-e7997f9fcb3e/istio-proxy/0.log"
Nov 01 10:32:48.770464 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:48.770387    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-797f57964-nlqd8_3415ad8e-a2dd-4804-9eff-7145c8fa8678/payload-processing/0.log"
Nov 01 10:32:49.899553 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:49.899506    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-785787c595-mh66b_f475e2f8-e3ef-41f4-9671-03d69acdd652/thanos-query/0.log"
Nov 01 10:32:50.320929 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:50.320899    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-785787c595-mh66b_f475e2f8-e3ef-41f4-9671-03d69acdd652/kube-rbac-proxy-web/0.log"
Nov 01 10:32:50.740790 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:50.740756    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-785787c595-mh66b_f475e2f8-e3ef-41f4-9671-03d69acdd652/kube-rbac-proxy/0.log"
Nov 01 10:32:51.161465 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:51.161392    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-785787c595-mh66b_f475e2f8-e3ef-41f4-9671-03d69acdd652/prom-label-proxy/0.log"
Nov 01 10:32:51.585349 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:51.585307    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-785787c595-mh66b_f475e2f8-e3ef-41f4-9671-03d69acdd652/kube-rbac-proxy-rules/0.log"
Nov 01 10:32:52.006256 ip-10-0-134-93 kubenswrapper[4314]: I1101 10:32:52.006226    4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-785787c595-mh66b_f475e2f8-e3ef-41f4-9671-03d69acdd652/kube-rbac-proxy-metrics/0.log"